Skip to content

Example: watch a page and show what is new

Markdown

This application reads a page of example.com when you run checkPage, in the browser of the person named by the run, with their session. It keeps the items it finds in its SQLite database, one set per person, and its home page lists them, newest first. It puts together what the other pages of the guide show one at a time: a domain reached through the browser, request.fetch from an action, its database and a home page.

The three files of the watch example

Section titled “The three files of the watch example”

manifest.json declares the site, the SQLite database, the home page and two actions: checkPage, which reads the page, and listNewItems, which the home page calls. Replace the slug with a name of your own, since a slug is unique on all of Apigami.

manifest.json
{
  "openapi": "3.1.0",
  "info": {
    "title": "Page watch",
    "description": "Reads a page of example.com on demand and lists the items that appeared on it.",
    "version": "1.0.0"
  },
  "x-apigami": {
    "manifest_version": 2,
    "slug": "acme-page-watch",
    "entrypoint": "main.ts",
    "domains": [{ "domain": "example.com", "via": "extension" }],
    "database": "sqlite",
    "pages": [{ "kind": "home", "id": "home", "title": "What is new", "page_url": "/" }]
  },
  "paths": {
    "/check": {
      "post": {
        "operationId": "checkPage",
        "summary": "Check the page",
        "description": "Reads the watched page in the person's browser and remembers the items it had not seen yet.",
        "responses": {
          "200": {
            "description": "How many items the page lists, and how many of them are new",
            "content": {
              "application/json": {
                "schema": {
                  "type": "object",
                  "required": ["checked", "new_items"],
                  "properties": {
                    "checked": { "type": "integer" },
                    "new_items": { "type": "integer" }
                  }
                },
                "example": { "checked": 12, "new_items": 2 }
              }
            }
          }
        }
      }
    },
    "/items": {
      "post": {
        "operationId": "listNewItems",
        "summary": "List the items seen",
        "description": "The 50 items most recently found on the page for the person viewing, newest first.",
        "responses": {
          "200": {
            "description": "The items, newest first",
            "content": {
              "application/json": {
                "schema": {
                  "type": "object",
                  "required": ["items"],
                  "properties": {
                    "items": {
                      "type": "array",
                      "items": {
                        "type": "object",
                        "required": ["url", "title", "first_seen_at"],
                        "properties": {
                          "url": { "type": "string" },
                          "title": { "type": "string" },
                          "first_seen_at": { "type": "string", "format": "date-time" }
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

main.ts exports default { fetch(request, env) }, which serves the home page and both actions. Each action learns the person from its own run token, and every row carries their user_ref. The database is env.DB: the actions create its table when it does not exist yet, since nothing may query it while the module loads.

main.ts
import home from './home.html' with { type: 'text' }


const WATCHED_PAGE = 'https://www.example.com/news'
const PAGE_HEADERS = {
  'content-type': 'text/html; charset=utf-8',
  'x-content-type-options': 'nosniff',
  'referrer-policy': 'no-referrer',
  'cross-origin-opener-policy': 'same-origin-allow-popups',
  'x-robots-tag': 'noindex',
}


const createTable = (env) => env.DB.prepare(`create table if not exists seen_items (
  user_ref text not null,
  url text not null,
  title text not null,
  first_seen_at text not null, -- ISO 8601 in UTC, so that sorting the text sorts the time
  primary key (user_ref, url)
)`).run()


// The person this run serves, from its own token: null for a run that serves nobody.
const personOf = async (env, token: string | null) => {
  if (!token) return null
  const me = await fetch(`${env.APIGAMI_API_URL}/v1/me`, { headers: { Authorization: `Bearer ${token}` } })
  return me.ok ? (await me.json()).user_ref : null
}


// Every link to an article under /news/. Adapt the pattern to the page you watch.
const itemsOf = (html: string) => [...html.matchAll(/<a[^>]+href="([^"#?]*\/news\/[^"#?]+)"[^>]*>([^<]+)<\/a>/g)]
  .map(([, href, title]) => ({ url: new URL(href, WATCHED_PAGE).href, title: title.trim().slice(0, 300) }))
  .filter((item) => item.title)


const checkPage = async (request: Request, env) => {
  const token = request.headers.get('X-Apigami-Run')
  const userRef = await personOf(env, token)
  if (!userRef) return Response.json({ error: 'this action reads the page in one person\'s browser, and its token names nobody' }, { status: 401 })


  const answer = await fetch(`${env.APIGAMI_API_URL}/call`, {
    method: 'POST',
    headers: {
      Authorization: `Bearer ${token}`,
      'Content-Type': 'application/json',
      Accept: 'application/octet-stream',
      'Accept-Encoding': 'gzip',
    },
    body: JSON.stringify({
      op: 'request.fetch',
      params: { domain: 'example.com', url: WATCHED_PAGE, responseType: 'binary' },
    }),
  })
  // Apigami's refusal (the browser offline, the site not approved) makes the run read `failed`, with its reason.
  if (!answer.ok) return Response.json({ error: `Apigami answered ${answer.status}: ${await answer.text()}` }, { status: 502 })
  const status = Number(answer.headers.get('X-Apigami-Status'))
  if (status !== 200) return Response.json({ error: `the page answered ${status}` }, { status: 502 })


  const items = itemsOf(await answer.text())
  if (!items.length) return Response.json({ checked: 0, new_items: 0 })
  await createTable(env)
  const insert = env.DB.prepare(`insert into seen_items (user_ref, url, title, first_seen_at) values (?1, ?2, ?3, ?4)
    on conflict do nothing returning url`)
  const firstSeenAt = new Date().toISOString()
  const inserted = await env.DB.batch(items.map((item) => insert.bind(userRef, item.url, item.title, firstSeenAt)))
  return Response.json({ checked: items.length, new_items: inserted.flatMap(({ results }) => results).length })
}


const listNewItems = async (request: Request, env) => {
  const userRef = await personOf(env, request.headers.get('X-Apigami-Run'))
  if (!userRef) return Response.json({ error: 'unauthorised' }, { status: 401 })
  await createTable(env)
  const { results: items } = await env.DB.prepare(`select url, title, first_seen_at from seen_items
    where user_ref = ?1 order by first_seen_at desc, url limit 50`).bind(userRef).all()
  return Response.json({ items })
}


export default {
  async fetch(request: Request, env) {
    const { pathname } = new URL(request.url)
    if (request.method === 'GET' && pathname === '/healthz') return Response.json({ ok: true })
    if (request.method === 'GET' && pathname === '/') return new Response(home, { headers: PAGE_HEADERS })
    if (request.method === 'POST' && pathname === '/check') return checkPage(request, env)
    if (request.method === 'POST' && pathname === '/items') return listNewItems(request, env)
    return Response.json({ error: 'not found' }, { status: 404 })
  },
}

home.html renders nothing personal on the server: the list comes from listNewItems, under the viewer’s token.

home.html
<!doctype html>
<html lang="en">
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>What is new</title>
<h1>What is new on example.com</h1>
<p id="status">Loading…</p>
<ul id="items"></ul>
<script src="https://api.eu.apigami.com/apigami.js"></script>
<script type="module">
  // A visitor who is not connected yet goes through Apigami's connect page first, and comes back here.
  const { items } = await apigami.call('listNewItems', {})
  document.getElementById('status').textContent = items.length
    ? `${items.length} items, newest first`
    : 'Nothing yet: run checkPage with your browser open to read the page.'
  document.getElementById('items').replaceChildren(...items.map((item) => {
    const link = Object.assign(document.createElement('a'), { href: item.url, textContent: item.title, target: '_blank', rel: 'noopener' })
    const line = document.createElement('li')
    line.append(link, ` — ${new Date(item.first_seen_at).toLocaleString()}`)
    return line
  }))
</script>

Put it to work

Section titled “Put it to work”
  1. Write the three files to a new application: app_settings with a name, then app_version_write, or over HTTP as the starter shows.
  2. Try it: app_run with action: "checkPage" and the extension_id of the Chrome that is signed in to the site. The first run connects you to the application, and that Chrome asks you to approve example.com. Over HTTP, connect first at https://app.apigami.com/connect?app_id=APP_ID in that Chrome, then call POST /v1/apps/{app_id}/actions/checkPage with your application’s key, X-Apigami-Version and X-Apigami-User-Ref set to the person’s user_ref (Try an action that reaches a site through the browser). The first run answers { "checked": 12, "new_items": 12 }: everything on the page is new once. The next ones count only what appeared since.
  3. app_deploy, app_run again with version on the deployed version, then app_publish (Run it, deploy it, publish it).

Run checkPage again to refresh the saved items. For recurring checks, declare a global trigger whose action adds one queued task per person, carrying their user_ref; the trigger itself has no person or browser. See Triggers.

To watch another site, change example.com in the manifest and in the request.fetch call, the WATCHED_PAGE address and the link pattern in itemsOf.