# Example: watch a page and show what is new

> A complete application in three files. An action reads a page through the person's browser, its database remembers the items seen, and a home page lists them.

This application reads a page of `example.com` when you run `checkPage`, in the browser of the person
named by the run, with their session. It keeps the items it finds in its SQLite database, one set per
person, and its home page lists them, newest first. It puts together what the other pages of the guide
show one at a time: [a domain reached through the browser](/docs/app-manifest/#domains-via-extension-or-direct),
[`request.fetch` from an action](/docs/app-server/#what-requestfetch-answers-your-code),
[its database](/docs/app-database/#its-database) and [a home page](/docs/app-pages/#the-home-page).

## The three files of the watch example

`manifest.json` declares the site, the SQLite database, the home page and two actions:
`checkPage`, which reads the page, and `listNewItems`, which the home page calls. Replace the `slug`
with a name of your own, since a slug is unique on all of Apigami.

```json title="manifest.json"
{
  "openapi": "3.1.0",
  "info": {
    "title": "Page watch",
    "description": "Reads a page of example.com on demand and lists the items that appeared on it.",
    "version": "1.0.0"
  },
  "x-apigami": {
    "manifest_version": 2,
    "slug": "acme-page-watch",
    "entrypoint": "main.ts",
    "domains": [{ "domain": "example.com", "via": "extension" }],
    "database": "sqlite",
    "pages": [{ "kind": "home", "id": "home", "title": "What is new", "page_url": "/" }]
  },
  "paths": {
    "/check": {
      "post": {
        "operationId": "checkPage",
        "summary": "Check the page",
        "description": "Reads the watched page in the person's browser and remembers the items it had not seen yet.",
        "responses": {
          "200": {
            "description": "How many items the page lists, and how many of them are new",
            "content": {
              "application/json": {
                "schema": {
                  "type": "object",
                  "required": ["checked", "new_items"],
                  "properties": {
                    "checked": { "type": "integer" },
                    "new_items": { "type": "integer" }
                  }
                },
                "example": { "checked": 12, "new_items": 2 }
              }
            }
          }
        }
      }
    },
    "/items": {
      "post": {
        "operationId": "listNewItems",
        "summary": "List the items seen",
        "description": "The 50 items most recently found on the page for the person viewing, newest first.",
        "responses": {
          "200": {
            "description": "The items, newest first",
            "content": {
              "application/json": {
                "schema": {
                  "type": "object",
                  "required": ["items"],
                  "properties": {
                    "items": {
                      "type": "array",
                      "items": {
                        "type": "object",
                        "required": ["url", "title", "first_seen_at"],
                        "properties": {
                          "url": { "type": "string" },
                          "title": { "type": "string" },
                          "first_seen_at": { "type": "string", "format": "date-time" }
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}
```

`main.ts` exports `default { fetch(request, env) }`, which serves the home page and both actions. Each
action learns the person from its own run token, and every row carries their `user_ref`. The database is
`env.DB`: the actions create its table when it does not exist yet, since nothing may query it while the
module loads.

```ts title="main.ts"
import home from './home.html' with { type: 'text' }

const WATCHED_PAGE = 'https://www.example.com/news'
const PAGE_HEADERS = {
  'content-type': 'text/html; charset=utf-8',
  'x-content-type-options': 'nosniff',
  'referrer-policy': 'no-referrer',
  'cross-origin-opener-policy': 'same-origin-allow-popups',
  'x-robots-tag': 'noindex',
}

const createTable = (env) => env.DB.prepare(`create table if not exists seen_items (
  user_ref text not null,
  url text not null,
  title text not null,
  first_seen_at text not null, -- ISO 8601 in UTC, so that sorting the text sorts the time
  primary key (user_ref, url)
)`).run()

// The person this run serves, from its own token: null for a run that serves nobody.
const personOf = async (env, token: string | null) => {
  if (!token) return null
  const me = await fetch(`${env.APIGAMI_API_URL}/v1/me`, { headers: { Authorization: `Bearer ${token}` } })
  return me.ok ? (await me.json()).user_ref : null
}

// Every link to an article under /news/. Adapt the pattern to the page you watch.
const itemsOf = (html: string) => [...html.matchAll(/<a[^>]+href="([^"#?]*\/news\/[^"#?]+)"[^>]*>([^<]+)<\/a>/g)]
  .map(([, href, title]) => ({ url: new URL(href, WATCHED_PAGE).href, title: title.trim().slice(0, 300) }))
  .filter((item) => item.title)

const checkPage = async (request: Request, env) => {
  const token = request.headers.get('X-Apigami-Run')
  const userRef = await personOf(env, token)
  if (!userRef) return Response.json({ error: 'this action reads the page in one person\'s browser, and its token names nobody' }, { status: 401 })

  const answer = await fetch(`${env.APIGAMI_API_URL}/call`, {
    method: 'POST',
    headers: {
      Authorization: `Bearer ${token}`,
      'Content-Type': 'application/json',
      Accept: 'application/octet-stream',
      'Accept-Encoding': 'gzip',
    },
    body: JSON.stringify({
      op: 'request.fetch',
      params: { domain: 'example.com', url: WATCHED_PAGE, responseType: 'binary' },
    }),
  })
  // Apigami's refusal (the browser offline, the site not approved) makes the run read `failed`, with its reason.
  if (!answer.ok) return Response.json({ error: `Apigami answered ${answer.status}: ${await answer.text()}` }, { status: 502 })
  const status = Number(answer.headers.get('X-Apigami-Status'))
  if (status !== 200) return Response.json({ error: `the page answered ${status}` }, { status: 502 })

  const items = itemsOf(await answer.text())
  if (!items.length) return Response.json({ checked: 0, new_items: 0 })
  await createTable(env)
  const insert = env.DB.prepare(`insert into seen_items (user_ref, url, title, first_seen_at) values (?1, ?2, ?3, ?4)
    on conflict do nothing returning url`)
  const firstSeenAt = new Date().toISOString()
  const inserted = await env.DB.batch(items.map((item) => insert.bind(userRef, item.url, item.title, firstSeenAt)))
  return Response.json({ checked: items.length, new_items: inserted.flatMap(({ results }) => results).length })
}

const listNewItems = async (request: Request, env) => {
  const userRef = await personOf(env, request.headers.get('X-Apigami-Run'))
  if (!userRef) return Response.json({ error: 'unauthorised' }, { status: 401 })
  await createTable(env)
  const { results: items } = await env.DB.prepare(`select url, title, first_seen_at from seen_items
    where user_ref = ?1 order by first_seen_at desc, url limit 50`).bind(userRef).all()
  return Response.json({ items })
}

export default {
  async fetch(request: Request, env) {
    const { pathname } = new URL(request.url)
    if (request.method === 'GET' && pathname === '/healthz') return Response.json({ ok: true })
    if (request.method === 'GET' && pathname === '/') return new Response(home, { headers: PAGE_HEADERS })
    if (request.method === 'POST' && pathname === '/check') return checkPage(request, env)
    if (request.method === 'POST' && pathname === '/items') return listNewItems(request, env)
    return Response.json({ error: 'not found' }, { status: 404 })
  },
}
```

`home.html` renders nothing personal on the server: the list comes from `listNewItems`, under the
viewer's token.

```html title="home.html"
<!doctype html>
<html lang="en">
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>What is new</title>
<h1>What is new on example.com</h1>
<p id="status">Loading…</p>
<ul id="items"></ul>
<script src="https://api.eu.apigami.com/apigami.js"></script>
<script type="module">
  // A visitor who is not connected yet goes through Apigami's connect page first, and comes back here.
  const { items } = await apigami.call('listNewItems', {})
  document.getElementById('status').textContent = items.length
    ? `${items.length} items, newest first`
    : 'Nothing yet: run checkPage with your browser open to read the page.'
  document.getElementById('items').replaceChildren(...items.map((item) => {
    const link = Object.assign(document.createElement('a'), { href: item.url, textContent: item.title, target: '_blank', rel: 'noopener' })
    const line = document.createElement('li')
    line.append(link, ` — ${new Date(item.first_seen_at).toLocaleString()}`)
    return line
  }))
</script>
```

## Put it to work

1. Write the three files to a new application: `app_settings` with a name, then `app_version_write`,
   or over HTTP as [the starter](/docs/build-an-application/#starter-template-hello-world-without-a-browser)
   shows.
2. Try it: `app_run` with `action: "checkPage"` and the `extension_id` of the Chrome that is signed in to
   the site. The first run connects you to the application, and that Chrome asks you to approve `example.com`.
   Over HTTP, connect first at `https://app.apigami.com/connect?app_id=APP_ID` in that Chrome, then call
   `POST /v1/apps/{app_id}/actions/checkPage` with your application's key, `X-Apigami-Version` and
   `X-Apigami-User-Ref` set to the person's `user_ref`
   ([Try an action that reaches a site through the browser](/docs/app-publish/#try-an-action-that-reaches-a-site-through-the-browser)).
   The first run answers
   `{ "checked": 12, "new_items": 12 }`: everything on the page is new once. The next ones count only
   what appeared since.
3. `app_deploy`, `app_run` again with `version` on the deployed version, then `app_publish`
   ([Run it, deploy it, publish it](/docs/app-publish/#run-it-deploy-it-publish-it)).

Run `checkPage` again to refresh the saved items. For recurring checks, declare a global trigger
whose action adds one queued task per person, carrying their `user_ref`; the trigger itself has no
person or browser. See [Triggers](/docs/app-database/#triggers-x-apigamicrons).

To watch another site, change `example.com` in the manifest and in the `request.fetch` call, the
`WATCHED_PAGE` address and the link pattern in `itemsOf`.