# indexnowkit for PHP Tell search engines (Bing, Yandex, Naver, Seznam, Yep, and the other participants of the IndexNow registry) about new, changed and deleted pages the moment a model is committed. One attribute on the model, one key, done. Google does not participate; IndexNow is a notification, not indexing. | Package | Framework | Install | |---|---|---| | [Symfony bundle](symfony-bundle/index.md) | Symfony 6.4 \| 7 \| 8 + Doctrine | `composer require indexnowkit/symfony-bundle indexnowkit/doctrine` | | [Laravel](laravel/index.md) | Laravel 12 \| 13 | `composer require indexnowkit/laravel` | | [Yii2](yii2/index.md) | Yii 2.0.45+ | `composer require indexnowkit/yii2` | | [Doctrine](doctrine/index.md) | Doctrine ORM without Symfony | `composer require indexnowkit/doctrine` | | [Core](core/index.md) | plain PHP, any PSR-18 client | `composer require indexnowkit/core` | | [Sitemap](sitemap/index.md) | the `sitemap` command of every adapter | `composer require indexnowkit/sitemap` | Start with the page of your framework, then the [attribute reference](core/attribute-reference.md), the [configuration](core/configuration.md) (one table for the three adapters) and the [operations guide](core/operations.md) with its production checklist. Machine-readable: [llms.txt](llms.txt), [llms-full.txt](llms-full.txt). Issues and pull requests: [github.com/indexnowkit/php](https://github.com/indexnowkit/php). # Symfony IndexNow bundle — `indexnowkit/symfony-bundle` Tell search engines about new, changed and deleted pages the moment a Doctrine entity is committed. One attribute on the entity, one env variable, done. [](https://packagist.org/packages/indexnowkit/symfony-bundle) [](https://packagist.org/packages/indexnowkit/symfony-bundle) [](https://github.com/indexnowkit/php/actions) [](https://github.com/indexnowkit/spec)    [](https://github.com/indexnowkit/php/blob/main/packages/symfony-bundle/LICENSE) [Русская версия](https://github.com/indexnowkit/php/blob/main/packages/symfony-bundle/README.ru.md) · Issues and pull requests: [github.com/indexnowkit/php](https://github.com/indexnowkit/php/issues) (the `php-*` repositories are read-only splits) ## Who gets notified **Yandex, Bing (and DuckDuckGo via Bing), Naver, Seznam, Yep, Internet Archive, Amazon** — every engine in the [IndexNow](https://www.indexnow.org) [registry](https://www.indexnow.org/searchengines.json). One request to the shared endpoint reaches all of them; name engines explicitly only to reach a single one. **Google: no.** Google does not support IndexNow, its sitemap ping endpoint is gone (404) and the Indexing API is restricted to `JobPosting` / `BroadcastEvent`. This bundle will not pretend otherwise. **Notification, not indexing.** IndexNow tells an engine that a URL changed; whether and when the page is crawled and indexed is the engine's decision. See the result in Bing Webmaster Tools (IndexNow Insights) and Yandex.Webmaster (Indexing → Reindex pages); a useful metric is the share of submitted URLs in the index after a few days. Deleted pages: answer 410 (gone for good) or 404 (temporarily); for a move answer 301 and submit both URLs; a soft-404 or a redirect to the home page does harm. Bing's URL Submission API and Google's Indexing API are different protocols and not covered here. ## Why this over X Most IndexNow packages are a thin HTTP client: you collect the URLs, you call it, you read the answer. This family does the part that goes wrong in practice: - **Declared on the model** (`#[IndexNow]`) and submitted from the ORM hooks — no controller code to forget. - **After the commit**, not on flush: a rolled-back transaction announces nothing. - **Debounce** (10 minutes per URL, shared through your cache), **batches** of up to 10 000 URLs, one key per host from env. - **Answers handled**: 202 (key pending), 422, 429 with `Retry-After` back-off and a retry through your queue, 403 escalation. - **`check` before the first submission** says what is wrong (key file, engines, queue, cache, environment); `explain` says why a URL was or was not sent. - **One core** under the Symfony, Laravel, Yii2 and Doctrine adapters with a shared conformance suite: the same behaviour everywhere, documented once. ## Install ```bash composer require indexnowkit/symfony-bundle composer require symfony/http-client nyholm/psr7 # any PSR-18 client works; this pair is auto-configured composer require indexnowkit/doctrine # for automatic submission when entities change composer require indexnowkit/sitemap # optional: the indexnow:sitemap command bin/console indexnow:key:generate --write-env # adds INDEXNOW_KEY to .env.local ``` The Flex recipe registers the bundle, creates `config/packages/indexnowkit.yaml` and imports the key file route. Without Flex, add `IndexNowKit\SymfonyBundle\IndexNowKitBundle` to `config/bundles.php` and import `@IndexNowKitBundle/config/routes.php` from `config/routes.yaml`. ```yaml # config/packages/indexnowkit.yaml indexnowkit: key: '%env(INDEXNOW_KEY)%' base_url: '%env(INDEXNOW_BASE_URL)%' # used by console commands and Messenger workers ``` Entity hooks need `indexnowkit/doctrine` **and** `doctrine/doctrine-bundle`. Without them the bundle still works for manual submission, and `indexnow:check` says so instead of failing silently. ## Declare what has a public page `#[IndexNow]` is repeatable: one attribute per family of public URLs the entity has. ```php use Doctrine\ORM\Mapping as ORM; use IndexNowKit\Attribute\{IndexNow, IndexNowDefaults}; #[ORM\Entity] #[IndexNowDefaults(when: 'isPublished', fields: ['slug', 'title', 'body', 'published'])] #[IndexNow(route: 'post_show', params: ['slug' => 'slug'])] #[IndexNow(route: 'post_amp', params: ['slug' => 'slug'], when: 'hasAmp')] #[IndexNow(via: 'category')] // a changed post also refreshes its category page #[IndexNow(urls: ['/'])] // and the homepage class Post { #[ORM\Id, ORM\GeneratedValue, ORM\Column] public ?int $id = null; #[ORM\ManyToOne] public ?Category $category = null; public function __construct( #[ORM\Column(unique: true)] public string $slug, #[ORM\Column] public string $title = '', #[ORM\Column(type: 'text')] public string $body = '', #[ORM\Column] public bool $published = true, #[ORM\Column] public bool $amp = false, ) {} public function isPublished(): bool { return $this->published; } public function hasAmp(): bool { return $this->amp; } } ``` | Option | Meaning | |---|---| | `route` / `params` | route name and `param => property, getter, "self", dotted.path` or a typed `Param\*` value | | `resolver` | a `UrlResolverInterface` service id or class for anything custom | | `via` | an accessor to a related object or collection whose pages are resubmitted | | `url` / `urls` | an accessor returning the URL(s), or literal URLs | | `when` / `whenFields` | bool accessor; unpublished entities are skipped and `published → draft` is sent as a deletion | | `fields` | for updates, submit only when one of these fields changed | | `events` | subset of `created`, `updated`, `deleted` | | `locales` | `current` (default), `all` (every `framework.enabled_locales`), or a list | | `host` | generate this rule's URLs on another host (multi-domain) | | `name` | stable rule id for logs, `indexnow:explain` and overriding in a subclass | Full model, typed parameters, inheritance and the semantics table: [core attribute reference](../core/attribute-reference.md). ## Verify ```bash bin/console indexnow:check # config, key file reachable, engines, dispatch, Doctrine hooks bin/console indexnow:check --live # also sends a real probe request to every engine ``` Run it after every key rotation and after every deployment that touches the configuration. It is the command that answers most "it does not work" reports on its own. ## How it works - URLs are collected in `onFlush` / `postFlush` and handed over **only after the outermost transaction commits** (a DBAL driver middleware watches the real COMMIT). Rolled-back changes are never submitted. - Every rule of an entity is classified separately: the article page can be an update while the AMP page of the same entity is a deletion, in the same flush. - Everything collected during one HTTP request, console command or Messenger message is sent as **one batch** after the response was sent (`kernel.terminate`), never inside your request. - `dispatch: auto` uses **Messenger** when a transport is configured, otherwise sends synchronously after the response. `sync` always sends on terminate. `none` collects and never sends, for applications that drain the collector themselves. - The same URL is not re-sent within **10 minutes** (`debounce.per_url`, stored in `cache.app`), batches are split at **10 000 URLs**, hosts are grouped, `202` is a success, `403` means the key file is wrong. - Failures are logged on the `indexnow` Monolog channel and never break your request. `http.timeout` (10 s) and `throttle.max_requests_per_minute` (60, per process) apply to the HTTP client the bundle builds on first use. ## Manual submission ```php public function __construct(private readonly IndexNowKit\IndexNowKit $indexNow) {} $this->indexNow->submit(['/posts/hello', 'https://www.example.com/about']); $this->indexNow->submitEntity($post); $this->indexNow->explain($post, IndexNowKit\Event::Updated); // which rule produced which URL ``` ## Commands | Command | Options | |---|---| | `indexnow:check` | `--live` send a real probe · `--host` check one host only · `--probe-url` page to probe when the root redirects | | `indexnow:submit ` | `-f, --force` ignore the debounce store · `--dry-run` · `--json` | | `indexnow:submit-entity [ids...]` | `--event=updated`, `created` or `deleted` · `--limit` (default 1000, when no ids) · `--explain` show rule → URL and send nothing · `-f, --force` · `--dry-run` · `--json` | | `indexnow:explain ` | `--event=updated`, `created` or `deleted` | | `indexnow:sitemap [sitemap]` | `--changed-since="1 day"` · `--allow-foreign-hosts` follow CDN-hosted parts · `-f, --force` · `--dry-run` list only · `--json` | | `indexnow:key:generate` | `-l, --length` (8-128, default 32) · `--alphanumeric` · `--write-env[=FILE]` (default `.env.local`) · `--force` rotate an existing key | `` accepts an FQCN or a short `App\Entity` name. `indexnow:submit-entity` and `indexnow:explain` need Doctrine. ### Sitemaps `composer require indexnowkit/sitemap # optional: the indexnow:sitemap command` `indexnow:sitemap` with no argument reads `sitemap.url`, else `/sitemap.xml`; a local path or `file://` URL reads the file without the web server. XML and text sitemaps, indexes and gzip are handled by the [`indexnowkit/sitemap`](../sitemap/index.md) package; the command streams and submits every `batch.max_urls` URLs, so size is not a concern. `sitemap.enabled: false` removes the command; decorating `indexnowkit.sitemap_reader` shapes what it submits ([docs/extending.md](extending.md)). Without the package everything else works unchanged: `indexnow:sitemap` says `indexnowkit/sitemap is not installed: composer require indexnowkit/sitemap` and exits 1, `indexnow:check` prints `sitemap: not installed (…)`, a `sitemap` block left in the yaml still compiles and is ignored. Nothing is logged about it. ## Configuration The full annotated tree, every default and every compile-time validation: [docs/configuration.md](configuration.md). | Topic | | |---|---| | Multiple domains | [docs/multi-domain.md](multi-domain.md) | | Async delivery and retries | [docs/messenger.md](messenger.md) | | HTTP client, proxy, scoped clients | [docs/http-client.md](http-client.md) | | Doctrine details, priorities, connections | [docs/doctrine.md](doctrine.md) | | Custom resolvers | [docs/custom-resolvers.md](custom-resolvers.md) | | Extending: what is replaceable, decorating services | [docs/extending.md](extending.md) | | Testing your integration | [docs/testing.md](testing.md) | | Troubleshooting | [docs/troubleshooting.md](troubleshooting.md) | ## Operations - [Production checklist](../core/operations.md#production-checklist) — key and base URL, `check` in the deploy pipeline, `strict_hosts`, a shared debounce store, a monitored queue, staging that cannot submit, the three lines to alert on. - [Monitoring rules and the Sentry filter](../core/operations.md#monitoring-rules), [deleted pages](../core/operations.md#deleted-pages-what-your-site-must-return), [what not to submit](../core/operations.md#what-not-to-submit). - [Multi-domain: hosts, www and apex, hreflang](multi-domain.md) · [troubleshooting](troubleshooting.md). ## Debugging Three tools, in the order you should reach for them. 1. **`bin/console indexnow:explain 'App\Entity\Post' 42`** walks the whole decision path for one entity — rules, event subscription, `when` guard, `fields` filter, resolved URLs, normalization, host and key, key file, debounce — and sends nothing. 2. **The Web Profiler panel** shows what the request collected, what was actually sent, and the HTTP outcome per engine, alongside the dispatch mode, the key file URL per host and the debounce window. 3. **The `indexnow` Monolog channel** carries everything. Set it to `debug` while diagnosing: the reason a rule decided *not* to produce a URL is logged there. Message texts and levels are listed in the [operations guide](../core/operations.md). An invalid configuration does not throw from a flush: IndexNow is disabled, one `critical` line is logged, and `indexnow:check` prints the exact error. ## Limitations - DQL and QueryBuilder bulk `UPDATE` / `DELETE` bypass the unit of work: use `indexnow:submit` or `$indexNow->submit()`. - Sub-domains are separate hosts: give each its own key with the `hosts` map, and set `strict_hosts: true` so a host you did not configure is skipped rather than announced under the default key. - `dispatch: sync` depends on `kernel.terminate` actually firing. An early `exit()`, a fatal error, or a worker runtime whose bridge does not dispatch it per request will discard the batch — with a warning. Under Swoole, RoadRunner or FrankenPHP prefer `dispatch: messenger`. - Long-running custom commands should call `$indexNow->flush()` periodically instead of accumulating URLs for the whole process lifetime. - Outside production (`production_environments`, default `prod`/`production`), a missing `INDEXNOW_KEY` switches `dry_run` on instead of failing, so dev and test never hit the real API. - A renamed page (changed slug) announces its old URL as deleted and the new one as updated in the same flush; an entity whose slug is a `readonly` property only gets the new URL (logged at `debug`). ## Compatibility Public API of the bundle: configuration nodes, command names and options, service ids and aliases listed in [docs/extending.md](extending.md), the `Console\*Interface`s of `indexnowkit/console` and the core's `Adapter\SubmitterFactoryInterface` they are aliased to, the Messenger message and handler, and the container parameters listed in [docs/configuration.md](configuration.md). `DependencyInjection\*` is wiring, not API. The core's rules apply, including the "may grow" interfaces: [bc.md](../core/bc.md); what this package itself keeps stable: [docs/bc.md](bc.md). Before 1.0 a minor version may break; every break is listed under "Changed" in [CHANGELOG.md](https://github.com/indexnowkit/php/blob/main/packages/symfony-bundle/CHANGELOG.md) with the migration. ## Notes for AI assistants - Composer package `indexnowkit/symfony-bundle` (Symfony 6.4 | 7 | 8, on `indexnowkit/core`); entity hooks need `indexnowkit/doctrine` + `doctrine/doctrine-bundle`; the `sitemap` command needs `indexnowkit/sitemap`. Configuration: `config/packages/indexnowkit.yaml`, root key `indexnowkit`. - Minimal complete snippet (every `use` included): ```php use Doctrine\ORM\Mapping as ORM; use IndexNowKit\Attribute\{IndexNow, IndexNowDefaults}; #[ORM\Entity] #[IndexNowDefaults(when: 'isPublished', fields: ['slug', 'title', 'published'])] #[IndexNow(route: 'post_show', params: ['slug' => 'slug'])] #[IndexNow(urls: ['/'])] class Post { /* ORM columns, isPublished() */ } ``` - Verify: `bin/console indexnow:check` (exit 1 on any error; `--strict` fails on warnings too, `--json` for machines), `bin/console indexnow:config --json` (the effective configuration, keys masked: paste it into a bug report), `bin/console indexnow:explain 'App\\Entity\\Post' 1` (why a URL was or was not produced), `bin/console indexnow:submit-entity 'App\\Entity\\Post' 1 --dry-run`. - Pitfalls: - `dispatch: auto` exists in Symfony (`auto` | `messenger` | `sync` | `none`) and Yii2 (`auto` | `queue` | `sync` | `none`), **not** in Laravel (`queue` | `sync` | `none`). - Locales: `router.locales` in Laravel, `router.languages` in Yii2, `framework.enabled_locales` in Symfony; `locales: 'all'` on a rule uses that list. - `url:` names an accessor (method or property) that returns the URL; `urls:` is a list of literal URLs. Never put a literal in `url:`. - A string in `when:` is an accessor read as truthy (`published`, `isPublished`). A status string needs `Equals`: `when: new Equals('status', 'published')` (`IndexNowKit\Attribute\Param\Equals`). - Manual submission is `submitEntity()` in Symfony, `submitModel()` in Laravel, `submitRecord()` in Yii2; the commands are `indexnow:submit-entity`, `indexnow:submit-model`, `indexnow/submit-record`. Bulk queries (`update()`, `DB::table()`, `updateAll()`) fire no hooks: submit afterwards with those. - Laravel has two classes called `IndexNowKit`: the facade `IndexNowKit\Laravel\Facades\IndexNowKit` and the core service `IndexNowKit\IndexNowKit` (inject by type). Yii2 exposes the core through `Yii::$app->indexnow->kit()`. - Outside production a configured key with `dry_run` unset makes `check` fail (a staging copy would submit real URLs): set `dry_run: true` there, or `dry_run: false` explicitly when it submits on purpose. - Unknown configuration keys are warned about at boot (typos such as debounce.per_urls); the key list is `Config::OPTIONS` plus the adapter's own keys. ## Other frameworks | | | |---|---| | PHP | [core](../core/index.md), [doctrine](../doctrine/index.md), [laravel](../laravel/index.md), [yii2](../yii2/index.md) | | JS/TS | @indexnowkit/core, next, prisma (soon) | | Python | indexnowkit, indexnowkit-django (soon) | Design rationale: [docs/spec](https://github.com/indexnowkit/spec). Changelog: [CHANGELOG.md](https://github.com/indexnowkit/php/blob/main/packages/symfony-bundle/CHANGELOG.md). MIT. IndexNow is a trademark of its owner; this project is independent and not affiliated with Microsoft, Yandex or indexnow.org. # Laravel IndexNow package — `indexnowkit/laravel` Tell search engines about new, changed and deleted pages the moment an Eloquent model is committed. One attribute on the model, one env variable, done. [](https://packagist.org/packages/indexnowkit/laravel) [](https://packagist.org/packages/indexnowkit/laravel) [](https://github.com/indexnowkit/php/actions) [](https://github.com/indexnowkit/spec)    [](https://github.com/indexnowkit/php/blob/main/packages/laravel/LICENSE) [Русская версия](https://github.com/indexnowkit/php/blob/main/packages/laravel/README.ru.md) · Issues and pull requests: [github.com/indexnowkit/php](https://github.com/indexnowkit/php/issues) (the `php-*` repositories are read-only splits) ## Who gets notified **Yandex, Bing (and DuckDuckGo via Bing), Naver, Seznam, Yep, Internet Archive, Amazon** — every engine in the [IndexNow](https://www.indexnow.org) [registry](https://www.indexnow.org/searchengines.json). One request to the shared endpoint reaches all of them; name engines explicitly only to reach a single one. **Google: no.** Google does not support IndexNow, its sitemap ping endpoint is gone (404) and the Indexing API is restricted to `JobPosting` / `BroadcastEvent`. This package will not pretend otherwise. **Notification, not indexing.** IndexNow tells an engine that a URL changed; whether and when the page is crawled and indexed is the engine's decision. See the result in Bing Webmaster Tools (IndexNow Insights) and Yandex.Webmaster (Indexing → Reindex pages); a useful metric is the share of submitted URLs in the index after a few days. Deleted pages: answer 410 (gone for good) or 404 (temporarily); for a move answer 301 and submit both URLs; a soft-404 or a redirect to the home page does harm. Bing's URL Submission API and Google's Indexing API are different protocols and not covered here. ## Why this over X Most IndexNow packages are a thin HTTP client: you collect the URLs, you call it, you read the answer. This family does the part that goes wrong in practice: - **Declared on the model** (`#[IndexNow]`) and submitted from the ORM hooks — no controller code to forget. - **After the commit**, not on flush: a rolled-back transaction announces nothing. - **Debounce** (10 minutes per URL, shared through your cache), **batches** of up to 10 000 URLs, one key per host from env. - **Answers handled**: 202 (key pending), 422, 429 with `Retry-After` back-off and a retry through your queue, 403 escalation. - **`check` before the first submission** says what is wrong (key file, engines, queue, cache, environment); `explain` says why a URL was or was not sent. - **One core** under the Symfony, Laravel, Yii2 and Doctrine adapters with a shared conformance suite: the same behaviour everywhere, documented once. ## Install ```bash composer require indexnowkit/laravel composer require indexnowkit/sitemap # optional: the indexnow:sitemap command php artisan vendor:publish --tag=indexnow-config # config/indexnow.php (optional, every key has a default) php artisan indexnow:key:generate --write-env # adds INDEXNOW_KEY to .env php artisan indexnow:check # config, key file reachable, queue, cache ``` The service provider is auto-discovered. Laravel ships Guzzle, which is the PSR-18 client the package discovers; any other PSR-18 client works too (`indexnow.http.client`). ```dotenv INDEXNOW_KEY=... # from key:generate INDEXNOW_BASE_URL=https://www.example.com # defaults to APP_URL; used by artisan and queue workers ``` ## Declare what has a public page `#[IndexNow]` is repeatable: one attribute per family of public URLs the model has. `IndexNowable` registers the observer. ```php use Illuminate\Database\Eloquent\Model; use Illuminate\Database\Eloquent\Relations\BelongsTo; use IndexNowKit\Attribute\{IndexNow, IndexNowDefaults}; use IndexNowKit\Laravel\Eloquent\IndexNowable; #[IndexNowDefaults(when: 'isPublished', fields: ['slug', 'title', 'body', 'published'])] #[IndexNow(route: 'posts.show', params: ['post' => 'self'])] // route model binding #[IndexNow(route: 'posts.amp', params: ['slug' => 'slug'], when: 'hasAmp')] #[IndexNow(via: 'category')] // a changed post also refreshes its category page #[IndexNow(urls: ['/'])] // and the homepage class Post extends Model { use IndexNowable; protected $casts = ['published' => 'bool', 'amp' => 'bool']; public function isPublished(): bool { return $this->published; } public function hasAmp(): bool { return $this->amp; } /** @return BelongsTo */ public function category(): BelongsTo { return $this->belongsTo(Category::class); } } ``` | Option | Meaning | |---|---| | `route` / `params` | route name and `param => attribute, method, "self", dotted.path` or a typed `Param\*` value | | `resolver` | a `UrlResolverInterface` class or container binding for anything custom | | `via` | a relation (or dotted path) whose pages are resubmitted | | `url` / `urls` | a method returning the URL(s), or literal URLs | | `when` / `whenFields` | bool attribute or method; drafts are skipped and `published → draft` is sent as a deletion | | `fields` | for updates, submit only when one of these attributes changed | | `events` | subset of `created`, `updated`, `deleted` | | `locales` | `current` (default), `all` (`indexnow.router.locales`), or a list | | `host` | generate this rule's URLs on another host (multi-domain) | | `name` | stable rule id for logs, `indexnow:explain` and overriding in a subclass | Accessors read Eloquent attributes, casts, accessors and relations (`category.slug`) and fall back to methods (`isPublished()`). `params: ['post' => 'self']` passes the model to `route()`, so `{post}` and `{post:slug}` both work. A `when` attribute that only has a **database** default is not on the model right after `create()`: give it a model default (`protected $attributes = ['published' => false]`). Full model, typed parameters, inheritance and the semantics table: [core attribute reference](../core/attribute-reference.md). ### Models you cannot annotate ```php // AppServiceProvider::boot() use IndexNowKit\Attribute\{IndexNow, IndexNowDefaults, RuleSet}; use IndexNowKit\Laravel\Facades\IndexNowKit; IndexNowKit::observe(Product::class, [new IndexNow(route: 'products.show', params: ['product' => 'self'])], new IndexNowDefaults(when: 'is_active')); IndexNowKit::rules()->registerFor(Page::class, fn (Page $page): ?RuleSet => ...); // decided per object ``` Two classes are called `IndexNowKit`. The **facade** `IndexNowKit\Laravel\Facades\IndexNowKit` (above) proxies the `IndexNowManager` of this package: `observe()`, `rules()`, `submitModel()`, `submitModels()`, `submit()`, `collect()`, `flush()`, `explain()`. The **core** `IndexNowKit\IndexNowKit` is the same service without the Eloquent-specific parts; inject it by type (`public function __construct(private IndexNowKit $indexNow)`) or take it from the facade with `IndexNowKit::kit()`. Import one of them per file, or alias the other. ## Verify ```bash php artisan indexnow:check # config, key file reachable, engines, queue connection, cache store, spool php artisan indexnow:check --live # also sends a real probe request to every engine ``` Run it after every key rotation and after every deployment that touches the configuration. ## How it works - Observer callbacks resolve URLs **while the old state is still live** (`getOriginal()` in `updated`, the row in `deleting`) and hand them over through `Connection::afterCommit()`: nothing leaves before the outermost transaction commits, a rolled-back transaction (or savepoint) discards them. `DB::transaction()` nesting is handled by Laravel's transaction manager. - Every rule is classified separately: the article page can be an update while the AMP page of the same model is a deletion, in the same request. - Everything collected during one request, artisan command or queue job is sent as **one batch** in `app()->terminating()` (or after each handled job), never inside your request. - `dispatch: queue` (the default) pushes a `SubmitUrlsJob`; 429 and 5xx are retried with backoff, `Retry-After` wins, 403/422 fail the job so a broken key file shows up in `failed_jobs`. `QUEUE_CONNECTION=sync` runs it inline. - `SoftDeletes`: soft delete is a deletion, `restore()` a creation, `forceDelete()` a deletion. - A renamed page (changed slug, or a changed route key behind `self`) announces its old URL as deleted and the new one as updated, in the same batch. - Nothing thrown from a rule, a resolver or the HTTP layer reaches your application: it is logged, the save succeeds. ## Commands | Command | Options | |---|---| | `indexnow:check` | `--live` real probe · `--host=` one host · `--probe-url=` page for the probe | | `indexnow:submit ` | `-f, --force` ignore debounce · `--dry-run` · `--json` | | `indexnow:submit-model [ids...]` | `--event=` · `--limit=` · `--explain` · `-f, --force` · `--dry-run` · `--json` | | `indexnow:explain ` | `--event=` — rules, `when`, URLs, key, debounce; sends nothing | | `indexnow:sitemap [sitemap]` | `--changed-since="1 day"` · `--allow-foreign-hosts` · `-f, --force` · `--dry-run` · `--json` | | `indexnow:key:generate` | `-l, --length` · `--alphanumeric` · `--write-env[=FILE]` (default `.env`) · `--force` rotate | `` accepts an FQCN or a short `App\Models` name. ### Sitemaps `composer require indexnowkit/sitemap # optional: the indexnow:sitemap command` `indexnow:sitemap` with no argument reads `indexnow.sitemap.url`, else `/sitemap.xml`; a local path works too. Schedule it: `Schedule::command('indexnow:sitemap --changed-since="1 day"')->daily()`. Without the package everything else works unchanged: `indexnow:sitemap` says `indexnowkit/sitemap is not installed: composer require indexnowkit/sitemap` and exits 1, `indexnow:check` prints `sitemap: not installed (…)`, the `sitemap` block of `config/indexnow.php` is ignored. Nothing is logged about it. Details: [docs/sitemap.md](sitemap.md). ## Configuration Every key of `config/indexnow.php`, its default and what it does: [docs/configuration.md](configuration.md). | Topic | | |---|---| | Queue, retries, Horizon | [docs/queue.md](queue.md) | | Multiple domains and locales | [docs/multi-domain.md](multi-domain.md) | | Sitemaps | [docs/sitemap.md](sitemap.md) | | Extending: bindings you can replace, custom resolvers, checks | [docs/extending.md](extending.md) | | Testing your integration | [docs/testing.md](testing.md) | | Troubleshooting | [docs/troubleshooting.md](troubleshooting.md) | ## Operations - [Production checklist](../core/operations.md#production-checklist) — key and base URL, `check` in the deploy pipeline, `strict_hosts`, a shared debounce store, a monitored queue, staging that cannot submit, the three lines to alert on. - [Monitoring rules and the Sentry filter](../core/operations.md#monitoring-rules), [deleted pages](../core/operations.md#deleted-pages-what-your-site-must-return), [what not to submit](../core/operations.md#what-not-to-submit). - [Multi-domain: hosts, www and apex, hreflang](multi-domain.md) · [queue](queue.md) · [troubleshooting](troubleshooting.md). ## Debugging 1. **`php artisan indexnow:explain "App\Models\Post" 42`** walks the decision path for one model — rules, event subscription, `when`, `fields`, resolved URLs, normalization, host and key, debounce — and sends nothing. 2. **The log channel** (`indexnow.logging.channel`, default channel otherwise) carries everything; at `debug` it also says why a rule decided *not* to produce a URL. Messages and levels: [operations guide](../core/operations.md). 3. **`failed_jobs`** holds batches an engine rejected permanently (403: key file not reachable). An invalid configuration does not throw from a save: IndexNow is disabled, one `critical` line is logged, and `indexnow:check` prints the exact error. ## Limitations - `Model::query()->update()`, `delete()`, `insert()`, `upsert()` and `DB::table()` fire no model events (conformance A13): call `IndexNowKit::submitModels($query->get())` or `php artisan indexnow:submit-model` afterwards. - `attach()` / `detach()` / `sync()` on a pivot fire no events on the owner. Put `$touches = ['posts']` on the related model: the owner's `updated` (only `updated_at` changed) reaches a rule without a `fields` filter. - `dispatch: sync` depends on `terminating` firing. Under Octane it does; an early `exit()` or a fatal error discards the batch with a warning. Prefer the default `queue`. - Sub-domains are separate hosts: give each its own key with the `hosts` map, and set `strict_hosts: true`. - Outside production (`production_environments`, default `prod`/`production`), a missing `INDEXNOW_KEY` switches `dry_run` on instead of failing. ## Compatibility Public API: `config/indexnow.php` keys, command names and options, the container bindings listed in [docs/extending.md](extending.md), `Facades\IndexNowKit` / `IndexNowManager`, `Eloquent\IndexNowable`, `Queue\SubmitUrlsJob`. The core's rules apply, including the "may grow" interfaces: [bc.md](../core/bc.md); what this package itself keeps stable: [docs/bc.md](bc.md). Before 1.0 a minor version may break; every break is listed under "Changed" in [CHANGELOG.md](https://github.com/indexnowkit/php/blob/main/packages/laravel/CHANGELOG.md) with the migration. Laravel 12 and 13, PHP 8.2–8.5 (Laravel 13 needs PHP 8.3). ## Notes for AI assistants - Composer package `indexnowkit/laravel` (Laravel 12 | 13, on `indexnowkit/core`); the `sitemap` command needs `indexnowkit/sitemap`. Configuration: `config/indexnow.php` and `INDEXNOW_*` env variables. - Minimal complete snippet (every `use` included): ```php use IndexNowKit\Attribute\{IndexNow, IndexNowDefaults}; use IndexNowKit\Laravel\Eloquent\IndexNowable; #[IndexNowDefaults(when: 'isPublished', fields: ['slug', 'title', 'published'])] #[IndexNow(route: 'posts.show', params: ['post' => 'self'])] // route model binding #[IndexNow(urls: ['/'])] class Post extends Model { use IndexNowable; public function isPublished(): bool { return (bool) $this->published; } } ``` - Verify: `php artisan indexnow:check` (exit 1 on any error; `--strict` fails on warnings too, `--json` for machines), `php artisan indexnow:config --json` (the effective configuration, keys masked: paste it into a bug report), `php artisan indexnow:explain 'App\\Models\\Post' 1` (why a URL was or was not produced), `php artisan indexnow:submit-model 'App\\Models\\Post' 1 --dry-run`. - Pitfalls: - `dispatch: auto` exists in Symfony (`auto` | `messenger` | `sync` | `none`) and Yii2 (`auto` | `queue` | `sync` | `none`), **not** in Laravel (`queue` | `sync` | `none`). - Locales: `router.locales` in Laravel, `router.languages` in Yii2, `framework.enabled_locales` in Symfony; `locales: 'all'` on a rule uses that list. - `url:` names an accessor (method or property) that returns the URL; `urls:` is a list of literal URLs. Never put a literal in `url:`. - A string in `when:` is an accessor read as truthy (`published`, `isPublished`). A status string needs `Equals`: `when: new Equals('status', 'published')` (`IndexNowKit\Attribute\Param\Equals`). - Manual submission is `submitEntity()` in Symfony, `submitModel()` in Laravel, `submitRecord()` in Yii2; the commands are `indexnow:submit-entity`, `indexnow:submit-model`, `indexnow/submit-record`. Bulk queries (`update()`, `DB::table()`, `updateAll()`) fire no hooks: submit afterwards with those. - Laravel has two classes called `IndexNowKit`: the facade `IndexNowKit\Laravel\Facades\IndexNowKit` and the core service `IndexNowKit\IndexNowKit` (inject by type). Yii2 exposes the core through `Yii::$app->indexnow->kit()`. - Outside production a configured key with `dry_run` unset makes `check` fail (a staging copy would submit real URLs): set `dry_run: true` there, or `dry_run: false` explicitly when it submits on purpose. - Unknown configuration keys are warned about at boot (typos such as debounce.per_urls); the key list is `Config::OPTIONS` plus the adapter's own keys. ## Other frameworks | | | |---|---| | PHP | [core](../core/index.md), [symfony-bundle](../symfony-bundle/index.md), [doctrine](../doctrine/index.md), [laravel](./index.md), [yii2](../yii2/index.md) | | JS/TS | @indexnowkit/core, next, prisma (soon) | | Python | indexnowkit, indexnowkit-django (soon) | Design rationale: [docs/spec](https://github.com/indexnowkit/spec). Changelog: [CHANGELOG.md](https://github.com/indexnowkit/php/blob/main/packages/laravel/CHANGELOG.md). MIT. IndexNow is a trademark of its owner; this project is independent and not affiliated with Microsoft, Yandex or indexnow.org. # Yii2 IndexNow extension — `indexnowkit/yii2` Tell search engines about new, changed and deleted pages the moment an ActiveRecord row is committed. One attribute on the model, one component, done. [](https://packagist.org/packages/indexnowkit/yii2) [](https://packagist.org/packages/indexnowkit/yii2) [](https://github.com/indexnowkit/php/actions) [](https://github.com/indexnowkit/spec)    [](https://github.com/indexnowkit/php/blob/main/packages/yii2/LICENSE) [Русская версия](https://github.com/indexnowkit/php/blob/main/packages/yii2/README.ru.md) · Issues and pull requests: [github.com/indexnowkit/php](https://github.com/indexnowkit/php/issues) (the `php-*` repositories are read-only splits) ## Who gets notified **Yandex, Bing (and DuckDuckGo via Bing), Naver, Seznam, Yep, Internet Archive, Amazon** — every engine in the [IndexNow](https://www.indexnow.org) [registry](https://www.indexnow.org/searchengines.json). One request to the shared endpoint reaches all of them; name engines explicitly only to reach a single one. **Google: no.** Google does not support IndexNow; this package will not pretend otherwise. **Notification, not indexing.** IndexNow tells an engine that a URL changed; whether and when the page is crawled and indexed is the engine's decision. See the result in Bing Webmaster Tools (IndexNow Insights) and Yandex.Webmaster (Indexing → Reindex pages); a useful metric is the share of submitted URLs in the index after a few days. Deleted pages: answer 410 (gone for good) or 404 (temporarily); for a move answer 301 and submit both URLs; a soft-404 or a redirect to the home page does harm. Bing's URL Submission API and Google's Indexing API are different protocols and not covered here. ## Why this over X Most IndexNow packages are a thin HTTP client: you collect the URLs, you call it, you read the answer. This family does the part that goes wrong in practice: - **Declared on the model** (`#[IndexNow]`) and submitted from the ORM hooks — no controller code to forget. - **After the commit**, not on flush: a rolled-back transaction announces nothing. - **Debounce** (10 minutes per URL, shared through your cache), **batches** of up to 10 000 URLs, one key per host from env. - **Answers handled**: 202 (key pending), 422, 429 with `Retry-After` back-off and a retry through your queue, 403 escalation. - **`check` before the first submission** says what is wrong (key file, engines, queue, cache, environment); `explain` says why a URL was or was not sent. - **One core** under the Symfony, Laravel, Yii2 and Doctrine adapters with a shared conformance suite: the same behaviour everywhere, documented once. ## Install ```bash composer require indexnowkit/yii2 symfony/http-client nyholm/psr7 # any PSR-18 client + PSR-17 factories work composer require indexnowkit/sitemap # optional: the indexnow/sitemap command ``` ```php // config/web.php and config/console.php 'bootstrap' => ['indexnow'], // registers the console controller and the key file route 'components' => [ 'indexnow' => [ 'class' => \IndexNowKit\Yii2\IndexNowComponent::class, 'options' => [ 'key' => getenv('INDEXNOW_KEY'), 'base_url' => 'https://www.example.com', // used by console commands and queue workers 'dry_run' => YII_ENV_DEV, // dev/staging: log the request, send nothing (check fails when this is unset outside production) ], ], ], ``` ```bash php yii indexnow/key-generate --write-env # writes INDEXNOW_KEY=… to .env (or prints the key) php yii indexnow/check # options, key file reachable, queue, cache, URL rules ``` Yii2 does not read `.env` by itself: export the variable (`export INDEXNOW_KEY=…`), put it in the web server or container environment, or load the file with `vlucas/phpdotenv` before `config/*.php` runs — `getenv('INDEXNOW_KEY')` returns `false` until one of these is done, and `check` says `no key configured`. In `yii2-app-basic`, `config/web.php` and `config/console.php` are independent: configure the `indexnow` component **and** `urlManager` (pretty URLs, rules) in both, or `check`, `explain` and `submit-record` see a different setup than the web application. Pretty URLs (`urlManager.enablePrettyUrl`) are required for the key file route `/.txt`. The package needs a PSR-18 client (`symfony/http-client` + `nyholm/psr7` as above, or Guzzle); it discovers one, or takes the component/class named in `http.client`. ## Declare what has a public page `#[IndexNow]` is repeatable: one attribute per family of public URLs. `IndexNowBehavior` registers the hooks. Save the example as `models/Post.php` under `namespace app\models;` — it reads the columns `slug`, `title`, `body`, `published`, `amp` (the AMP page exists while it is true) and `category_id`; `Category` is a record of your own with its own `#[IndexNow]` rule (drop the `via: 'category'` line if you have none). ```php use IndexNowKit\Attribute\{IndexNow, IndexNowDefaults}; use IndexNowKit\Yii2\ActiveRecord\IndexNowBehavior; use yii\db\ActiveQuery; use yii\db\ActiveRecord; #[IndexNowDefaults(when: 'published', fields: ['slug', 'title', 'body', 'published'])] #[IndexNow(route: 'post/view', params: ['slug' => 'slug'])] #[IndexNow(route: 'post/amp', params: ['slug' => 'slug'], when: 'amp')] #[IndexNow(via: 'category')] // a changed post also refreshes its category page #[IndexNow(urls: ['/'])] // and the homepage final class Post extends ActiveRecord { public static function tableName(): string { return 'posts'; } public function init(): void { parent::init(); $this->loadDefaultValues(); // `published` has a database default: make it visible before the first save } public function behaviors(): array { return [IndexNowBehavior::class]; } public function getCategory(): ActiveQuery { return $this->hasOne(Category::class, ['id' => 'category_id']); } } ``` | Option | Meaning | |---|---| | `route` / `params` | a Yii route (`controller/action`) and `param => attribute, method, "self", dotted.path` (`self` = the primary key) | | `resolver` | a `UrlResolverInterface` class or component id for anything custom | | `via` | a relation (or dotted path) whose pages are resubmitted | | `url` / `urls` | a method returning the URL(s), or literal URLs | | `when` / `whenFields` | bool attribute or method; drafts are skipped and `published → draft` is sent as a deletion | | `fields` | for updates, submit only when one of these attributes changed | | `events`, `locales`, `host`, `name` | subset of events; `current`/`all`/list (`router.languages`); another host; stable rule id | Accessors read ActiveRecord attributes and relations (`category.slug`) and fall back to methods. A `when` column that only has a **database** default is null on a fresh record: call `$this->loadDefaultValues()` in `init()` or set the attribute before `save()`. Classes you cannot annotate: `'active_record' => ['models' => [Product::class]]` in the options, or `Yii::$app->indexnow->observe(Product::class, [new IndexNow(...)])` at runtime. Full model, typed parameters, inheritance and the semantics table: [core attribute reference](../core/attribute-reference.md). ## How it works - URLs are resolved **in the ActiveRecord event**, while the old state is live (`changedAttributes` on `afterUpdate`, the row and its relations in `beforeDelete`). A renamed page announces its old URL as deleted. - Outside a transaction they go to the request collector right away. Inside one, Yii2 gives no savepoint events, so they are held with a verifier and **re-read by primary key when the transaction commits**: a change the row does not show (an inner `beginTransaction()` that rolled back) is dropped with every URL it produced. A rollback drops everything. One `SELECT` per changed record, only inside explicit transactions. Details: [docs/commit-safety.md](commit-safety.md). - Everything collected during one request is sent **after the response** (`Response::EVENT_AFTER_SEND`), in one batch; console commands flush when they end, queue workers after every job. - `dispatch: auto` (default) pushes a `SubmitUrlsJob` to the `queue` component when `yiisoft/yii2-queue` is configured (429/5xx re-pushed with the delay of `retry.*`, `Retry-After` honoured), else sends synchronously. Details: [docs/queue.md](queue.md). - Nothing thrown from a rule, a resolver or the HTTP layer reaches your application: it is logged under the `indexnow` category, the save succeeds. An invalid configuration disables IndexNow with one `critical` line; `php yii indexnow/check` prints the exact error. ## Commands | Command | Options | |---|---| | `indexnow/check` | `--live` real probe · `--host=` one host · `--probe-url=` page for the probe | | `indexnow/submit ` | `--force` ignore debounce · `--dry-run` · `--json` | | `indexnow/submit-record [ids...]` | `--event=` · `--limit=` · `--explain` · `--force` · `--dry-run` · `--json` | | `indexnow/explain ` | `--event=` — rules, `when`, URLs, key, debounce; sends nothing | | `indexnow/sitemap [sitemap]` | `--changed-since="1 day"` · `--allow-foreign-hosts` · `--force` · `--dry-run` · `--json` | | `indexnow/key-generate` | `--length` · `--alphanumeric` · `--write-env[=FILE]` · `--force` rotate | `` is an FQCN or a short name under `app\models`. Ids are space- or comma-separated. ### Sitemaps `composer require indexnowkit/sitemap # optional: the indexnow/sitemap command` `indexnow/sitemap` with no argument reads `sitemap.url`, else `/sitemap.xml`; a local path works too. Without the package everything else works unchanged: `indexnow/sitemap` says `indexnowkit/sitemap is not installed: composer require indexnowkit/sitemap` and exits 1, `indexnow/check` prints `sitemap: not installed (…)`, a `sitemap` block in the options is ignored, `sitemapConfig()` / `sitemapSource()` throw a `LogicException` with the same sentence. Nothing is logged about it. ## Configuration and docs Every option, its default and what it does: [docs/configuration.md](configuration.md). Commit safety: [docs/commit-safety.md](commit-safety.md). Replacing pieces, custom resolvers, checks: [docs/extending.md](extending.md). Queue, retries, failures: [docs/queue.md](queue.md). Several hosts, www and apex, languages: [docs/multi-domain.md](multi-domain.md). Testing your integration: [docs/testing.md](testing.md). ## Operations - [Production checklist](../core/operations.md#production-checklist) — key and base URL, `check` in the deploy pipeline, `strict_hosts`, a shared debounce store, a monitored queue, staging that cannot submit, the three lines to alert on. - [Monitoring rules and the Sentry filter](../core/operations.md#monitoring-rules), [deleted pages](../core/operations.md#deleted-pages-what-your-site-must-return), [what not to submit](../core/operations.md#what-not-to-submit). - [Multi-domain: hosts, www and apex, languages](multi-domain.md) · [queue](queue.md) · [commit safety](commit-safety.md) · [troubleshooting](troubleshooting.md). ## Debugging `php yii indexnow/check` validates the options, fetches the key file and reports how submissions are wired (queue, cache, pretty URLs, ActiveRecord hooks, sitemap spool); `php yii indexnow/explain 'app\models\Post' 1` shows the rules, guards and URLs of one record without sending anything; the `indexnow` log category at `debug` tells why a URL was or was not submitted. Symptoms and fixes: [docs/troubleshooting.md](troubleshooting.md). ## Limitations - `updateAll()`, `deleteAll()`, `updateAttributes()`, `updateCounters()` fire no events (conformance A13): call `Yii::$app->indexnow->submitRecords(Post::find()->where(...)->all())` or `php yii indexnow/submit-record` afterwards. - `link()` / `unlink()` write the junction row with a plain command, no event on the owner: save the owner with a bumped timestamp afterwards (`$post->updated_at = time(); $post->save(false)`), or call `submitRecord($post)`. - The sync driver of `yii2-queue` ignores the delay between attempts: 429/5xx attempts run back-to-back (development only, `check` warns). - Without pretty URLs the key file cannot be routed: enable them, or serve `/.txt` as a static file and set `key_file.enabled: false`. ## Compatibility Public API: the `options` tree, command names and options, `IndexNowComponent` methods and properties, `ActiveRecord\IndexNowBehavior`, `Queue\SubmitUrlsJob`. The core's rules apply: [bc.md](../core/bc.md); what this package itself keeps stable: [docs/bc.md](bc.md). Before 1.0 a minor version may break; every break is listed under "Changed" in [CHANGELOG.md](https://github.com/indexnowkit/php/blob/main/packages/yii2/CHANGELOG.md). Yii 2.0.45+, PHP 8.2–8.5. ## Notes for AI assistants - Composer package `indexnowkit/yii2` (Yii 2.0.45+, on `indexnowkit/core`); the `sitemap` command needs `indexnowkit/sitemap`. Configuration: the `indexnow` application component (`options` array), `'bootstrap' => ['indexnow']`. - Minimal complete snippet (every `use` included): ```php use IndexNowKit\Attribute\{IndexNow, IndexNowDefaults}; use IndexNowKit\Yii2\ActiveRecord\IndexNowBehavior; #[IndexNowDefaults(when: 'published', fields: ['slug', 'title', 'published'])] #[IndexNow(route: 'post/view', params: ['slug' => 'slug'])] #[IndexNow(urls: ['/'])] final class Post extends ActiveRecord { public function behaviors(): array { return [IndexNowBehavior::class]; } } ``` - Verify: `php yii indexnow/check` (exit 1 on any error; `--strict` fails on warnings too, `--json` for machines), `php yii indexnow/config --json` (the effective configuration, keys masked: paste it into a bug report), `php yii indexnow/explain 'app\\models\\Post' 1` (why a URL was or was not produced), `php yii indexnow/submit-record 'app\\models\\Post' 1 --dry-run`. - Pitfalls: - `dispatch: auto` exists in Symfony (`auto` | `messenger` | `sync` | `none`) and Yii2 (`auto` | `queue` | `sync` | `none`), **not** in Laravel (`queue` | `sync` | `none`). - Locales: `router.locales` in Laravel, `router.languages` in Yii2, `framework.enabled_locales` in Symfony; `locales: 'all'` on a rule uses that list. - `url:` names an accessor (method or property) that returns the URL; `urls:` is a list of literal URLs. Never put a literal in `url:`. - A string in `when:` is an accessor read as truthy (`published`, `isPublished`). A status string needs `Equals`: `when: new Equals('status', 'published')` (`IndexNowKit\Attribute\Param\Equals`). - Manual submission is `submitEntity()` in Symfony, `submitModel()` in Laravel, `submitRecord()` in Yii2; the commands are `indexnow:submit-entity`, `indexnow:submit-model`, `indexnow/submit-record`. Bulk queries (`update()`, `DB::table()`, `updateAll()`) fire no hooks: submit afterwards with those. - Laravel has two classes called `IndexNowKit`: the facade `IndexNowKit\Laravel\Facades\IndexNowKit` and the core service `IndexNowKit\IndexNowKit` (inject by type). Yii2 exposes the core through `Yii::$app->indexnow->kit()`. - Outside production a configured key with `dry_run` unset makes `check` fail (a staging copy would submit real URLs): set `dry_run: true` there, or `dry_run: false` explicitly when it submits on purpose. - Unknown configuration keys are warned about at boot (typos such as debounce.per_urls); the key list is `Config::OPTIONS` plus the adapter's own keys. ## Other frameworks | | | |---|---| | PHP | [core](../core/index.md), [symfony-bundle](../symfony-bundle/index.md), [doctrine](../doctrine/index.md), [laravel](../laravel/index.md) | | JS/TS | @indexnowkit/core, next, prisma (soon) | | Python | indexnowkit, indexnowkit-django (soon) | MIT. IndexNow is a trademark of its owner; this project is independent and not affiliated with Microsoft, Yandex or indexnow.org. # IndexNow client for PHP — `indexnowkit/core` Tell Yandex, Bing and the other [IndexNow](https://www.indexnow.org) engines which URLs changed, from any PHP application. Batching, debounce, throttling, retry policy, key file handling and the `#[IndexNow]` rule model, on top of PSR-18 / PSR-17 / PSR-3 / PSR-16 only. The framework adapters ([Symfony](../symfony-bundle/index.md), [Doctrine](../doctrine/index.md), [Laravel](../laravel/index.md), [Yii2](../yii2/index.md)) and the add-on packages build on it; use it directly in plain PHP, a CMS plugin or a custom framework. [](https://packagist.org/packages/indexnowkit/core) [](https://packagist.org/packages/indexnowkit/core) [](https://github.com/indexnowkit/php/actions) [](https://github.com/indexnowkit/spec)    [](https://github.com/indexnowkit/php/blob/main/packages/core/LICENSE) [Русская версия](https://github.com/indexnowkit/php/blob/main/packages/core/README.ru.md) · Issues and pull requests: [github.com/indexnowkit/php](https://github.com/indexnowkit/php/issues) (the `php-*` repositories are read-only splits) ## Who gets notified **Yandex, Bing (and DuckDuckGo via Bing), Naver, Seznam, Yep, Internet Archive, Amazon** — every engine in the IndexNow [registry](https://www.indexnow.org/searchengines.json). One request to the shared endpoint `api.indexnow.org` reaches all of them; name engines explicitly (`engines: [yandex, bing]`) only to reach a single one. Internet Archive has no working direct endpoint at the time of writing — it is reached through `api`. **Google: no.** Google does not support IndexNow, its sitemap ping endpoint is gone and the Indexing API is limited to `JobPosting` / `BroadcastEvent`. Keep your sitemap for Google; this library will not pretend otherwise. **Notification, not indexing.** IndexNow tells an engine that a URL changed; whether and when the page is crawled and indexed is the engine's decision. See the result in Bing Webmaster Tools (IndexNow Insights) and Yandex.Webmaster (Indexing → Reindex pages); a useful metric is the share of submitted URLs in the index after a few days. Deleted pages: answer 410 (gone for good) or 404 (temporarily); for a move answer 301 and submit both URLs; a soft-404 or a redirect to the home page does harm. Bing's URL Submission API and Google's Indexing API are different protocols and not covered here. ## Why this over X Most IndexNow packages are a thin HTTP client: you collect the URLs, you call it, you read the answer. This family does the part that goes wrong in practice: - **Declared on the model** (`#[IndexNow]`) and submitted from the ORM hooks — no controller code to forget. - **After the commit**, not on flush: a rolled-back transaction announces nothing. - **Debounce** (10 minutes per URL, shared through your cache), **batches** of up to 10 000 URLs, one key per host from env. - **Answers handled**: 202 (key pending), 422, 429 with `Retry-After` back-off and a retry through your queue, 403 escalation. - **`check` before the first submission** says what is wrong (key file, engines, queue, cache, environment); `explain` says why a URL was or was not sent. - **One core** under the Symfony, Laravel, Yii2 and Doctrine adapters with a shared conformance suite: the same behaviour everywhere, documented once. ## Install ```bash composer require indexnowkit/core symfony/http-client nyholm/psr7 # any PSR-18 client + PSR-17 factories work ``` If you use a framework, prefer its adapter: it wires everything below through your container and hooks into entity changes. The family: | Package | What | |---|---| | `indexnowkit/core` | this package: protocol client, rules, key file, the adapter kit | | [`indexnowkit/doctrine`](../doctrine/index.md) | Doctrine ORM listener plus a DBAL middleware, commit-safe | | [`indexnowkit/symfony-bundle`](../symfony-bundle/index.md) | Symfony: config, Messenger, key file route, commands, profiler panel | | [`indexnowkit/laravel`](../laravel/index.md) | Laravel: Eloquent observer, queue, key file route, artisan commands | | [`indexnowkit/yii2`](../yii2/index.md) | Yii2: ActiveRecord events with verify-on-commit, yii2-queue, console controller | | [`indexnowkit/sitemap`](../sitemap/index.md) | reads a sitemap (index, gzip, text) and submits its URLs; the `sitemap` command of every adapter | | [`indexnowkit/console`](../console/index.md) | the bodies of the `check`, `submit`, `submit-`, `explain`, `key:generate` commands and their definitions (`symfony/console`); every adapter requires it | | [`indexnowkit/testing`](../testing/index.md) | `require-dev`: the conformance kits (C01–C22, A01–A21), the H01–H05 assertions, the mock IndexNow server | ## Quick start ```php use IndexNowKit\Config; use IndexNowKit\IndexNowKit; $indexNow = IndexNowKit::create(Config::fromEnv()); // INDEXNOW_KEY, INDEXNOW_BASE_URL, ... foreach ($indexNow->submit(['/posts/hello', 'https://www.example.com/about']) as $result) { printf("%s %s %d %s\n", $result->engine, $result->status->value, $result->httpCode ?? 0, $result->error ?? ''); } ``` ```dotenv INDEXNOW_KEY=6f3c9a... # 8-128 characters, [A-Za-z0-9-] INDEXNOW_BASE_URL=https://www.example.com ``` `submit()` never throws for remote problems: every engine × host × batch yields a `Result` and a log line, and URLs that were not sent (debounced, disabled, dry-run, unknown host) yield a `skipped` result that says why. ## The key file Search engines verify ownership by fetching `https://{host}/{key}.txt`, whose body must be exactly the key. ```php $key = IndexNowKit\Key\KeyGenerator::generate(); // 32 hex characters, CSPRNG file_put_contents("public/$key.txt", $key); // or answer the request yourself: $body = (new KeyFileResponder($indexNow->keys))->bodyForPath($path, $host); // null -> 404 ``` Serve it with `200 OK` and `text/plain`, without redirects; `KeyFileResponder::headers()` has the right headers. A key file elsewhere on the host is fine with `key_location`. `Check\Checker` validates the configuration, fetches every key file and, with `liveProbe: true`, sends a real probe. `403` always means the key file is wrong; rotation guidance is in [docs/operations.md](operations.md). ## What happens to a URL 1. **Normalize** — relative paths resolved against `base_url`, scheme and host lower-cased, IDN hosts to punycode, default ports and fragments removed, dot-segments resolved. Anything that is not a public `http(s)` URL is dropped with a warning. 2. **De-duplicate** within the call, then **debounce**: URLs sent successfully in the last `debounce.per_url` seconds are skipped. A failing store never blocks delivery, it just stops de-duplicating and logs a warning. 3. **Group by host** and look up the key. Hosts without a key are `skipped` and never sent under another host's key. 4. **Chunk** into at most `batch.max_urls` URLs, **throttle** one token per HTTP request, and **POST** one batch per endpoint: `{"host", "key", "keyLocation"?, "urlList"}` as `application/json; charset=utf-8`. 5. **Interpret** the answer into a `Result` and mark successful URLs in the debounce store. ## Results | `status` | HTTP | `reason` | `retryable` | Meaning | |---|---|---|---|---| | `ok` | 200 | — | no | accepted | | `pending` | 202 | — | no | accepted, key verification pending; counts as success | | `failed` | 400 | `invalid_request` | no | malformed request (bug: please report) | | `failed` | 403 | `invalid_key` | no | key file not reachable or does not match | | `failed` | 422 | `unprocessable` | no | URLs do not belong to the host / `keyLocation` invalid | | `failed` | 429 | `rate_limited` | yes | `retryAfter` filled when the engine said so | | `failed` | 5xx | `server_error` | yes | | | `failed` | — | `transport` | yes | network failure or timeout | | `failed` | — or other | `unexpected` | see below | a misbehaving HTTP client (retryable) or a status no engine should return (not) | | `skipped` | — | `disabled` `dry_run` `debounced` `no_key` `invalid_url` | no | nothing was sent | `Reason` is the stable identifier for metrics and alerts, `Result::$error` the human sentence; `Reason::translationKey()` (`indexnowkit.reason.`) names the message for a UI. Decide whether to retry from `Result::$retryable`, not from the reason. `Result` also carries `engine`, `endpoint`, `host`, `urls`, `httpCode` and `metricLabels()`; `Result::retryableUrls($results)` collects what is worth retrying. ```php $indexNow->submitter->addListener(fn (IndexNowKit\Result $r) => $metrics->increment('indexnow_results_total', $r->metricLabels())); ``` Log lines go to the PSR-3 logger you pass to `IndexNowKit::create()`. See [docs/operations.md](operations.md) for the levels, the exact messages and a "my URL was not submitted" checklist. ## Declaring pages: `#[IndexNow]` `#[IndexNow]` is **repeatable**: one attribute per family of public URLs the object has. Exactly one source per rule — `route`, `resolver`, `via`, `url` or `urls`. Class-wide policy goes to `#[IndexNowDefaults]`, whose `when` is ANDed with each rule's own `when` (a draft page is never public, whatever the rule says). ```php use IndexNowKit\Attribute\{IndexNow, IndexNowDefaults, IndexNowUrl}; use IndexNowKit\Attribute\Param\{Accessor, Call, Formatted, Placeholder, Value}; #[IndexNowDefaults(when: 'isPublished', fields: ['slug', 'title', 'body', 'published'])] #[IndexNow(route: 'post_show', params: ['slug' => 'slug'])] // the article page #[IndexNow(route: 'post_amp', params: ['slug' => 'slug'], when: 'hasAmp', whenFields: ['ampEnabled'])] #[IndexNow(via: 'category')] // resubmit the category page #[IndexNow(via: 'tags')] // and every tag page #[IndexNow(urls: ['/', '/blog'])] // and two literal URLs class Post {} ``` Typed parameter sources, next to the plain accessor string (property, getter, `is`/`has` method, `dotted.path`, `self`): ```php #[IndexNow(route: 'post_show', params: [ 'year' => new Formatted('publishedAt', 'Y'), // DateTimeInterface::format() 'cat' => 'category.slug', // dotted path through a relation 'section' => new Value('blog'), // a constant 'slug' => new Call('slugFor', Placeholder::Locale), // a method call, one URL per locale ])] ``` Other shapes, all real cases: ```php #[IndexNow(url: 'publicUrl')] // a property or method returning string|iterable|null #[IndexNow(resolver: SyliusChannelUrls::class)] // a UrlResolverInterface class or service id #[IndexNow(route: 'page_show', params: ['slug' => 'slug'], host: new Accessor('tenant.domain'))] // multi-domain #[IndexNow(route: 'post_show', params: ['slug' => 'slug'], locales: 'all')] // localized routes class Page {} class Offer { #[IndexNowUrl(when: 'isLive')] // the get_absolute_url() convention public function getPublicUrl(): string { return '/offers/' . $this->code; } } ``` Rules are inherited from parent classes and identified by `name` (derived from the source, or given explicitly): a subclass rule whose name repeats an ancestor's **replaces** it, a new name **adds** a page. ### Deletion semantics Visibility (`when`) is evaluated per rule, before and after a change. `true → false` submits that rule's URLs as a **deletion** so engines recrawl the 404; `false → true` is a creation; no transition is an update filtered by `fields`. Deleting an object whose rule does not apply submits nothing: the page was never public. `when` is often a getter (`isPublished`) while the ORM change set holds the field (`published`). The convention `isPublished → published`/`is_published` and `getStatus → status` is applied automatically; when the names are unrelated, name the backing fields with `whenFields`. A status string or enum is not a boolean: use `when: new Equals('status', 'published')` (`IndexNowKit\Attribute\Param\Equals`); rules registered at runtime may pass a closure. Full model, semantics table and the adapter-facing types (`UrlRule`, `RuleSet`, `RuleRegistry`): [docs/attribute-reference.md](attribute-reference.md). ```php $indexNow = IndexNowKit::create($config, resolver: new AttributeUrlResolver(new AttributeReader(), $router, $locator)); $indexNow->submitEntity($post, IndexNowKit\Event::Updated); $urls = $indexNow->urlsFor($post, Event::Deleted); // resolve without sending $rows = $indexNow->explain($post, Event::Updated); // ResolvedUrl: which rule produced which URL ``` `urlsFor()`, `explain()` and `submitEntity()` go through `GuardedUrlResolver`, which never throws: an invalid attribute is logged and yields no URLs, so a typo cannot break a flush. ## Configuration | Option | Env | Default | Meaning | |---|---|---|---| | `enabled` | `INDEXNOW_ENABLED` | `true` | `false` drops every submission (logged at `info`) | | `key` | `INDEXNOW_KEY` | — | default key, used for every host not listed in `hosts` | | `hosts` | `INDEXNOW_HOSTS` (`a.com=KEY1,b.com=KEY2`) | `[]` | per-host `{key, key_location, base_url}` | | `strict_hosts` | `INDEXNOW_STRICT_HOSTS` | `false` | apply the default key only to the `base_url` host | | `base_url` | `INDEXNOW_BASE_URL` | `null` | resolves relative URLs; required outside HTTP requests | | `engines` | `INDEXNOW_ENGINES` | `['api']` | engine names or custom `https://` endpoints | | `dispatch` | `INDEXNOW_DISPATCH` | `sync` | adapter-defined delivery mode; the core only reports it | | `batch.max_urls` | `INDEXNOW_BATCH_MAX_URLS` | `10000` | URLs per request: the protocol's ceiling, not a target | | `debounce.per_url` | `INDEXNOW_DEBOUNCE_PER_URL` | `600` | seconds before the same URL is sent again (`0` = off) | | `throttle.max_requests_per_minute` | `INDEXNOW_THROTTLE_PER_MINUTE` | `60` | per-process request rate (`0` = unlimited) | | `http.timeout` | `INDEXNOW_HTTP_TIMEOUT` | `10.0` | seconds, applied to clients created by discovery | | `dry_run` | `INDEXNOW_DRY_RUN` | `false` | log the request instead of sending it | | `environment` | `INDEXNOW_ENV` / `APP_ENV` | — | anything but `prod`/`production` without a key turns `dry_run` on | Also `key_file.enabled`, `http.user_agent` and `key_location`. Every value is validated at construction, so a bad setup fails at boot, not at the first submission. Full reference, per-host overrides, `Config::with()`, `Config::OPTIONS` and `unknownOptions()`: [docs/configuration.md](configuration.md). ## Retries, queues and bulk No retries inside a web request: `429`/`5xx` come back as `retryable` results. Use `RetryingSubmitter` in CLI, cron and workers, or re-enqueue `Result::retryableUrls($results)` after `(new RetryPolicy())->delayAfter($results, $attempt)` seconds. Collect during a unit of work, deliver once: ```php $indexNow->collect(['/posts/1', '/posts/2']); // anywhere during the request $indexNow->flush(); // at the end of the unit of work ``` See [docs/retries-and-queues.md](retries-and-queues.md) for the worker recipe and bulk/migration guidance. Re-announcing a bulk change from the site's own URL list is the job of the add-on package in the family table (Install); `$kit->transport` is the transport such consumers read through. Adapters prove their wiring with `Testing\Conformance\CoreConformanceTestCase`: extend it, return the facade your container built and its `FakeTransport`, and the protocol scenarios of the spec run against it. ## Testing `IndexNowKit\Testing` is part of the published package: `FakeTransport` (records POSTs, answers queued responses), `ArrayLogger`, `FrozenClock`, `RecordingDispatcher`. ```php $transport = new FakeTransport(); $indexNow = IndexNowKit::create($config, transport: $transport, debounce: new NullDebounceStore()); $indexNow->submitEntity($post); self::assertSame(['https://www.example.com/posts/hello'], $transport->posts[0]['body']['urlList']); ``` More recipes in [docs/testing.md](testing.md). ## Extension points | Interface | Default | Replace it to | |---|---|---| | `Http\TransportInterface` | `Psr18Transport::discover()` | use your own HTTP stack (`LazyTransport` defers building it) | | `Key\KeyProviderInterface` | `StaticKeyProvider` | keys from a database, per tenant | | `Url\UrlNormalizerInterface` | `UrlNormalizer` | strip tracking parameters, enforce trailing slashes, map hosts | | `Url\UrlResolverInterface` | `NullUrlResolver` — build an `AttributeUrlResolver` and pass it as `resolver:` | turn objects into URLs your way | | `Url\RouteUrlResolverInterface` | — (adapter-provided) | bridge your framework's router | | `Attribute\AttributeReaderInterface` | `AttributeReader` | `RuleRegistry` for runtime rules, or your own metadata source | | `Collector\CollectorInterface` | `Collector` | a durable outbox, a per-tenant buffer | | `Debounce\DebounceStoreInterface` | `MemoryDebounceStore` | `Psr16DebounceStore`, or your own | | `Throttle\ThrottleInterface` | `TokenBucket` | `NullThrottle`, a shared limiter | | `Dispatch\DispatcherInterface` | `SyncDispatcher` | `CallableDispatcher` for a queue, `NullDispatcher` | | `SubmitterInterface` | `Submitter` | decorate (`RetryingSubmitter`), record, mock | Pass any of them to `IndexNowKit::create()` by name, or assemble the graph by hand: `Client` → `Submitter` → `Collector` + `DispatcherInterface` → `IndexNowKit`. The pieces a framework adapter wires from its configuration have factories with one source of error texts — `Http\TransportFactory::lazy()` (`http.client`), `Debounce\DebounceStoreFactory::fromConfig()` (`debounce.store`), `Dispatch\DispatcherFactory::fromConfig()` (`dispatch`), `fromConfig()` on `Collector`, `TokenBucket`, `AttributeUrlResolver` and `KeyFileResponder` — and `Adapter\ConfigFactory` turns a raw framework array into a `Config` without ever throwing from a hook. A container that assembles at runtime describes the whole graph once with `Adapter\ServicesBuilder` and gets it lazily from `Adapter\Services`; ORM hooks share `Hook\ObserverHelper`, queue jobs `Retry\WorkerOutcome`, commands the runners and `Console\Definitions` of `indexnowkit/console`. Writing an adapter? [docs/adapters.md](adapters.md). ## Exceptions All exceptions implement `IndexNowKit\Exception\IndexNowException`: `ConfigurationException` (invalid `Config`, attribute or resolver setup), `InvalidUrlException` (a URL that cannot be submitted, caught by `Submitter` and dropped with a warning), `InvalidArgumentException` (programming errors) and `Http\Exception\TransportException` (network failure, turned into a retryable `Result` by `Client`; consumers reading documents through the transport see it; `Checker` turns it into an error line). Nothing throws out of a lifecycle hook — see the error contract in [docs/adapters.md](adapters.md). ## Limitations - The same URL is not re-sent within `debounce.per_url` (10 minutes by default): that is what Yandex asks for. - No retries inside a web request; `TokenBucket` throttles per process. Multi-process limits belong to your queue. - Only `http(s)` URLs on hosts you hold a key for. Sub-domains are separate hosts, each with its own key file. - Bulk ORM operations bypass entity hooks in every adapter: submit those URLs yourself. - Google is not reachable through IndexNow. ## Requirements PHP 8.2+, `ext-json`, `ext-filter`, a PSR-18 client with PSR-17 factories (`symfony/http-client` and Guzzle are configured automatically with the timeout and no redirects; other clients are used as is). Optional: `ext-intl` (IDN via UTS #46, otherwise a pure-PHP punycode encoder). ## Versioning SemVer. Before 1.0, minor versions may contain breaking changes; they are listed in [CHANGELOG.md](https://github.com/indexnowkit/php/blob/main/packages/core/CHANGELOG.md). What is covered by the promise and what is not: [docs/bc.md](bc.md). ## Notes for AI assistants - Composer package `indexnowkit/core` (PHP ^8.2, PSR-18 client + PSR-17 factories needed). Framework users want an adapter instead: `indexnowkit/symfony-bundle`, `indexnowkit/laravel`, `indexnowkit/yii2`, `indexnowkit/doctrine`. - Minimal complete snippet (every `use` included): ```php use IndexNowKit\Config; use IndexNowKit\IndexNowKit; $indexNow = IndexNowKit::create(Config::fromEnv()); // INDEXNOW_KEY, INDEXNOW_BASE_URL, INDEXNOW_DRY_RUN $results = $indexNow->submit(['/posts/hello']); // list, never throws for remote errors ``` - Verify: `(new IndexNowKit\Check\Checker($config, $indexNow->keys, $indexNow->transport))->run()` is what the adapters' `check` command runs; `$indexNow->explain($object)` shows the rule behind every URL; every remote outcome is a `Result` with `status`/`reason`, nothing throws. - Pitfalls: - `dispatch: auto` exists in Symfony (`auto` | `messenger` | `sync` | `none`) and Yii2 (`auto` | `queue` | `sync` | `none`), **not** in Laravel (`queue` | `sync` | `none`). - Locales: `router.locales` in Laravel, `router.languages` in Yii2, `framework.enabled_locales` in Symfony; `locales: 'all'` on a rule uses that list. - `url:` names an accessor (method or property) that returns the URL; `urls:` is a list of literal URLs. Never put a literal in `url:`. - A string in `when:` is an accessor read as truthy (`published`, `isPublished`). A status string needs `Equals`: `when: new Equals('status', 'published')` (`IndexNowKit\Attribute\Param\Equals`). - Manual submission is `submitEntity()` in Symfony, `submitModel()` in Laravel, `submitRecord()` in Yii2; the commands are `indexnow:submit-entity`, `indexnow:submit-model`, `indexnow/submit-record`. Bulk queries (`update()`, `DB::table()`, `updateAll()`) fire no hooks: submit afterwards with those. - Laravel has two classes called `IndexNowKit`: the facade `IndexNowKit\Laravel\Facades\IndexNowKit` and the core service `IndexNowKit\IndexNowKit` (inject by type). Yii2 exposes the core through `Yii::$app->indexnow->kit()`. - Outside production a configured key with `dry_run` unset makes `check` fail (a staging copy would submit real URLs): set `dry_run: true` there, or `dry_run: false` explicitly when it submits on purpose. - Unknown configuration keys are warned about at boot (typos such as debounce.per_urls); the key list is `Config::OPTIONS` plus the adapter's own keys. ## Other packages | | | |---|---| | PHP | the family table under [Install](#install) | | JS/TS | `@indexnowkit/core`, `next`, `prisma` (planned) | | Python | `indexnowkit`, `indexnowkit-django` (planned) | Design rationale and the cross-language model: [docs/spec](https://github.com/indexnowkit/spec). Conformance suite: [indexnowkit/spec](https://github.com/indexnowkit/spec). MIT. IndexNow is a trademark of its owner; this project is independent and not affiliated with Microsoft, Yandex or indexnow.org. # Attribute reference [Русская версия](attribute-reference.ru.md) A class declares a **list of rules**, one per family of public URLs it has. PHP writes them as attributes; every other language in the family writes decorators or config objects. All of them compile down to the same `IndexNowKit\Attribute\UrlRule`, and everything downstream — event classification, guards, locales, `via` delegation, deduplication, `explain` output — consumes only that. ## The three attributes | Attribute | Target | Purpose | |---|---|---| | `#[IndexNow]` | class, **repeatable** | one URL rule | | `#[IndexNowDefaults]` | class | policy shared by every rule of the class and its subclasses | | `#[IndexNowUrl]` | public method | the method's return value is a URL family (the `get_absolute_url()` convention) | ```php use IndexNowKit\Attribute\{IndexNow, IndexNowDefaults, IndexNowUrl}; #[IndexNowDefaults(when: 'isPublished', fields: ['slug', 'title', 'body', 'published'])] #[IndexNow(route: 'post_show', params: ['slug' => 'slug'])] #[IndexNow(route: 'post_amp', params: ['slug' => 'slug'], when: 'hasAmp', whenFields: ['ampEnabled'])] class Post {} ``` ## Sources Exactly one source per `#[IndexNow]`. Zero sources, or two, throws `ConfigurationException` at compile time with a message naming the offenders. | Source | Value | Produces | |---|---|---| | `route` | framework route name | one URL per locale, generated by the adapter's `RouteUrlResolverInterface` | | `resolver` | `UrlResolverInterface` class name or service id | whatever the resolver returns | | `via` | accessor to a related object or collection | the related objects' own URLs, resolved as updates | | `url` | accessor returning `string`, `iterable` or `null` | those URLs | | `urls` | list of literal URLs | those URLs, absolute or `base_url`-relative | `url` and `urls` are easy to swap, so both are checked: `url: '/about'` and `urls: ['aboutUrl']` are rejected with a message telling you which one you meant. `resolver` needs a `ResolverLocatorInterface`. In plain PHP that is `ArrayResolverLocator`, which also instantiates a class name on demand as long as its constructor takes no required arguments. Framework adapters look the id up in the container. ## Parameters `params` maps a route parameter name to a source. A plain string is the accessor DSL; anything else is one of four typed `Param\ParamValue` objects. ### The accessor DSL Resolved in this order, on the object itself: 1. `'self'` — the object (route model binding: `params: ['post' => 'self']`); 2. a dotted path — each segment resolved recursively (`'category.slug'`); a non-object segment throws; 3. a method with that exact name; 4. `get`, `is` or `has` plus the capitalised name (`'published'` finds `getPublished()`, then `isPublished()`, then `hasPublished()`); 5. a property, including a private one. Nothing matched throws `ConfigurationException` naming the accessor and the class. ### Typed sources | Class | Example | Meaning | |---|---|---| | `Param\Accessor` | `new Accessor('category.slug')` | the explicit form of a plain accessor string | | `Param\Value` | `new Value('html')` | a constant | | `Param\Formatted` | `new Formatted('publishedAt', 'Y')` | `DateTimeInterface::format()` of the accessor's value | | `Param\Call` | `new Call('slugFor', Placeholder::Locale)` | a method call; extra arguments are passed as given | `Param\Placeholder::Locale` and `Param\Placeholder::Host` are substituted per generated URL, so a `Call` can return a per-locale slug or a per-tenant path. Extraction runs once per URL, not once per rule. ### Coercion Route parameters must be usable in a URL. The extractor accepts `null` and scalars as they are, unwraps a `BackedEnum` to its `value`, casts a `Stringable` value object to string, and passes plain objects through for route model binding. A bare `DateTimeInterface` is rejected with a message pointing at `new Formatted(...)`, because formatting a date implicitly is how a URL silently changes shape. Anything else throws. ## Rule options | Option | Type | Default | Meaning | |---|---|---|---| | `when` | accessor name, a `Condition` (`new Equals(path, value)` or your own), or a closure `fn(object): bool` (runtime rules only) | inherit | the page exists only while the condition holds | | `whenFields` | list of field names | `[]` | fields backing this rule's own `when` when its name does not match the field (a class-level `when` has its own `whenFields` in `#[IndexNowDefaults]`) | | `fields` | list of field names, or `null` | inherit, then `[]` | for updates only: submit when one of these changed; `[]` = any field | | `events` | subset of `created`, `updated`, `deleted` (strings or `Event` cases), or `null` | inherit, then all three | which lifecycle events the rule listens to | | `locales` | `'current'`, `'all'` or a list, or `null` | inherit, then `'current'` | locale expansion for localized routes | | `host` | string or `ParamValue` | `null` | generate this rule's URLs on that host (multi-domain) | | `name` | string | derived | stable rule id for logs, `explain` output and subclass overrides | `when` is a **conjunction**: the class-level `when` and the rule's own `when` must both hold. `fields`, `events` and `locales` are defaults a rule overrides; `null` means inherit, `[]` means "no filter". An accessor string is checked for truthiness, which is right for booleans and wrong for a status string (`'draft'` is truthy). For string or enum states use `Equals`, which also gives exact old-state detection from the ORM change set: ```php use IndexNowKit\Attribute\Param\Equals; #[IndexNow(route: 'post_show', params: ['slug' => 'slug'], when: new Equals('status', 'published'))] #[IndexNow(route: 'job_show', params: ['id' => 'id'], when: new Equals('state', JobState::Open))] // BackedEnum or its value ``` Rules registered at runtime (`RuleRegistry`) may pass a closure: `when: fn (WP_Post $p): bool => $p->post_status === 'publish'`. A closure's old value cannot be reconstructed, so list the fields it reads in `whenFields`; a change of one of them is treated as a visibility flip (see the semantics table). ### Your own conditions `Equals` is an `Attribute\Param\Condition` — `evaluate(object $subject): bool` — and any class implementing it goes in `when` the same way (an attribute argument must be a constant expression, so a condition class with a constructor of scalars, not a closure): ```php use IndexNowKit\Attribute\Param\Condition; use IndexNowKit\Attribute\Param\FieldCondition; final readonly class Between implements Condition { public function __construct(private string $path, private int $min, private int $max) {} public function evaluate(object $subject): bool { $value = ParamExtractor::read($subject, $this->path); return is_int($value) && $value >= $this->min && $value <= $this->max; } } final readonly class OneOf implements FieldCondition // reads one field: the classifier sees the old state { /** @param list $values */ public function __construct(private string $path, private array $values) {} public function evaluate(object $subject): bool { return $this->heldFor(ParamExtractor::read($subject, $this->path)); } public function field(): string { return $this->path; } public function heldFor(mixed $oldValue): bool { return in_array($oldValue, $this->values, true); } } #[IndexNow(route: 'offer_show', params: ['id' => 'id'], when: new OneOf('state', ['open', 'reserved']))] ``` A plain `Condition` has no old value: `ChangeClassifier` evaluates it on the current object, so `open → closed` is classified as a plain update, not as the deletion it is — unless `whenFields` names the field the condition reads (then a change of that field counts as a flip). Implement `FieldCondition` (`field()`, `heldFor($oldValue)`) when the condition reads one field, and the change set gives the exact old state, as it does for `Equals`. `Condition` and `FieldCondition` are in the Implement tier of [bc.md](bc.md), with the pre-1.0 caveat that they are new in 0.8. `Equals` is a condition, not a value source: `params: ['status' => new Equals(...)]` is a type error, and `ParamExtractor` names the fix. `explain` prints every condition with the value it read (`when: status ("draft") -> true — a non-empty string is truthy; use new Equals('status', "draft")`), and `explain --json` gives the same walk as a document. An unknown event name throws `ConfigurationException` naming the attribute and the value. ### Rule names Derived from the source when not given: the route name; `resolver:`; `via:`; `url:` (and `url:` for `#[IndexNowUrl]`); `urls:`. Two rules of the same class that would derive the same name get `#2`, `#3` appended in declaration order. Give an explicit `name` whenever you intend a subclass to override a specific rule, or whenever the derived name would be unstable. ## Class defaults and inheritance The compiler walks the class hierarchy **root first**, then the leaf. - `#[IndexNowDefaults]` merges field by field, the nearest declaration wins. A declaration that sets its own `when` also replaces the inherited `whenFields`; one that does not adds to them. - Rules accumulate. A rule whose name repeats an ancestor's **replaces** it; a new name **adds** a page. That is how a subclass changes one page without restating the others. - `#[IndexNowUrl]` is read on public methods declared by each class in the chain, so an override in a subclass wins. The method must not require arguments. - Interfaces and traits are **not** scanned: PHP does not inherit class attributes through them, and Doctrine mapping behaves the same way. ```php #[IndexNowDefaults(when: 'isPublished')] #[IndexNow(route: 'content_show', params: ['slug' => 'slug'])] abstract class Content {} #[IndexNow(route: 'content_show', params: ['slug' => 'slug', 'section' => new Value('news')])] // replaces #[IndexNow(route: 'news_amp', params: ['slug' => 'slug'])] // adds class News extends Content {} ``` A hierarchy where only some subclasses have public pages should carry no rules on the base class: what is not declared is not inherited. ## Semantics: event, before, after Visibility is evaluated per rule. `W` is the conjunction of the class `when` and the rule's `when`; `W_before` is reconstructed from the ORM change set. | ORM event | `W_before` | `W_after` | `fields` match | Rule event | State the URL is built from | |---|---|---|---|---|---| | insert | — | true | — | `Created` if subscribed | new state, after the write (ids assigned) | | insert | — | false | — | none | — | | update | true | true | yes | `Updated` if subscribed | new state, after the write | | update | true | true | no | none | — | | update | true | false | ignored | **`Deleted`** if subscribed | current state, before the write | | update | false | true | ignored | `Created` if subscribed | new state, after the write | | update | false | false | — | none | — | | delete | — | true | — | `Deleted` if subscribed | pre-delete state | | delete | — | false | — | none | — | | `via` target, any event | — | true | per target rule | target resolved as `Updated` | the target's own rules and guards | | `via` target, any event | — | false | — | none | — | Two consequences worth stating out loud. `fields` never suppresses a visibility transition, only a plain update: a page that just went dark is announced whichever field did it. And deleting an object whose rule does not apply submits nothing — that page was never public, so purging drafts stays quiet. ### Reconstructing `W_before` `ChangeClassifier::classify(UrlRule $rule, object $subject, array $changedFields, array $changeSet = [])` returns the `Event` a rule cares about, or `null`. Old-state visibility is best effort, in three tiers: 1. A `when` accessor whose backing field is present in the change set is evaluated **exactly** from the old value. The backing field is found by name, then by convention: `isPublished → published → is_published`, `hasAmp → amp → has_amp`, `getStatus → status`. `UrlRule::fieldCandidates()` exposes that list. 2. An accessor with no change-set entry, but a field it depends on (its candidates, or a declared `whenFields` entry) among the changed fields, is assumed to have **flipped**. A false positive costs one request; a false negative leaves a dead page in the index. 3. Otherwise the accessor keeps its current value. Name the fields with `whenFields` when the accessor is a method whose name has nothing to do with the column, for example `when: 'isVisibleToPublic', whenFields: ['status', 'visibleFrom']`. ## `via` `via` resubmits a related object's pages: a changed comment refreshes its post, a changed product refreshes its category. Targets are always resolved as `Updated`, because their page exists regardless of what happened to the source. Depth is capped at 3 and fan-out at 100 related objects per rule (constructor arguments `maxViaDepth` and `maxViaFanout` of `AttributeUrlResolver`); exceeding the depth throws, exceeding the fan-out logs a warning and stops. A target rule that delegates back through the same accessor name is skipped, so `A -> B -> A` terminates. Resulting URLs keep the whole chain in their rule name: `via:category -> category_show`. ## Field names `fields` and `whenFields` are **model field names** as the developer writes them, never database columns. Doctrine's `getEntityChangeSet()` gives exactly those. A declared field matches a changed one when they are equal or one is a dotted prefix of the other, so `fields: ['address']` catches an embeddable change reported as `address.city`. ## Types adapters consume ```php final readonly class UrlRule { public string $name; public RuleSource $source; // Route|Resolver|Via|Url|Urls public ?string $route; public array $params; public ?string $resolver; public ?string $via; public ?string $url; public array $urls; public array $when; public array $whenFields; public array $fields; public array $events; public array|string $locales; public string|ParamValue|null $host; public function listensTo(Event $event): bool; public function caresAbout(array $changedFields): bool; public function appliesTo(object $subject): bool; // every `when` accessor is truthy public function whenDependsOn(string $field): bool; public static function fieldCandidates(string $accessor): array; } ``` `RuleSet` is every rule of one class in declaration order (parents first). It is `Countable` and iterable, empty for classes without rules so callers never branch on null, and offers `isEmpty()`, `get(string $name)` and `listensTo(Event $event)` as a cheap pre-filter for ORM hooks. `AttributeReaderInterface::rules(string|object $classOrObject): RuleSet` is the lookup. The default `AttributeReader` compiles attributes through `RuleCompiler` and caches per class for the process lifetime. It throws `ConfigurationException` on a malformed declaration — ORM hooks must read through `ObjectChangeHandler` or `GuardedUrlResolver`, which log instead. `ResolvedUrl` carries provenance for `explain` output, logs and profiler panels: `url`, `rule`, `class`, `event`, `locale`, plus `source()` (`App\Entity\Post#post_amp`) and `ResolvedUrl::urls()` to flatten a list to deduplicated strings. ## Rules registered at runtime Models that cannot carry attributes — CMS post types, classes you do not own, a closure API — use `RuleRegistry`, which implements `AttributeReaderInterface` on top of an inner reader (attributes by default). ```php use IndexNowKit\Attribute\{IndexNow, IndexNowDefaults, RuleRegistry}; $registry = new RuleRegistry(); $registry->register(Post::class, [ new IndexNow(route: 'posts.show', params: ['post' => 'self']), new IndexNow(urls: ['/']), ], new IndexNowDefaults(when: 'isPublished')); $registry->register(WP_Post::class, [new IndexNow(resolver: 'wp_permalink')], new IndexNowDefaults( when: fn (WP_Post $post): bool => $post->post_status === 'publish', // or: new Equals('post_status', 'publish') whenFields: ['post_status'], )); $registry->registerFor(CmsPage::class, fn (CmsPage $page): ?RuleSet => $page->isSystem() ? null : $rulesFor($page)); $indexNow = IndexNowKit::create($config, attributes: $registry); ``` `register()` compiles attribute instances built in code, with no reflection. `registerFor()` decides per object and may return `null` to fall through to the inner reader. Registered rules replace whatever the inner reader would return for that class, and subclasses inherit them. ## Resolving without an ORM `AttributeUrlResolver` resolves every rule of a class through its source; `GuardedUrlResolver` wraps it so nothing throws. `ObjectChangeHandler` is the piece ORM hooks build on: it classifies a created, updated or deleted object per rule and resolves the URLs, logging every silent outcome. See [adapters.md](adapters.md). ## Anti-patterns Five declarations that compile, run, and submit the wrong thing. **1. A literal URL in `url:`.** `url:` names an accessor; `urls:` lists literals. ```php #[IndexNow(url: '/')] // wrong: reads a property or method called "/" #[IndexNow(urls: ['/'])] // right #[IndexNow(url: 'canonicalUrl')] // right: $post->canonicalUrl() or ->canonicalUrl ``` **2. A status string in `when:`.** A string is an accessor read as truthy, so `'published'` means "the attribute `published` is truthy" — a `status` column holding `'draft'` is truthy too. `explain` shows the value it read and says so. ```php #[IndexNow(route: 'post_show', params: ['slug' => 'slug'], when: 'status')] // wrong: 'draft' is truthy #[IndexNow(route: 'post_show', params: ['slug' => 'slug'], when: new Equals('status', 'published'))] // right #[IndexNow(route: 'post_show', params: ['slug' => 'slug', 'v' => new Equals('status', 'published')])] // wrong: a condition is not a param value (type error) ``` **2b. A custom `Condition` on a field that flips.** `when: new Published()` that reads `status` internally cannot tell the classifier what the old state was: `published → draft` is an update, the dead page stays indexed. Implement `FieldCondition`, or name the field in `whenFields`. **3. A rule on a page the engine must not index.** A preview, an admin page, a page with `noindex` or a `robots.txt` disallow: the engine fetches it, finds it unindexable, and counts a mistake against the key. ```php #[IndexNow(route: 'post_preview', params: ['slug' => 'slug'])] // wrong: preview pages carry noindex #[IndexNow(route: 'post_show', params: ['slug' => 'slug'], when: 'isPublished')] // right: the public page, guarded ``` **4. Non-canonical URLs.** Filter and sort variants, tracking parameters, the apex next to `www`, `http` next to `https`: submit the canonical page once. Generate URLs through the router with `base_url` on the canonical origin; do not build them by string concatenation from the request. ```php #[IndexNow(urls: ['/products?sort=price&utm_source=indexnow'])] // wrong: a variant, and a tracking parameter #[IndexNow(route: 'products_index')] // right: the canonical listing ``` **5. No `when` on a model that has drafts.** Without a guard every save submits, drafts included, and a page taken down is announced as an update, not a deletion. ```php #[IndexNow(route: 'post_show', params: ['slug' => 'slug'])] // wrong when Post has a draft state #[IndexNowDefaults(when: 'isPublished')] // right: drafts are skipped, #[IndexNow(route: 'post_show', params: ['slug' => 'slug'])] // published → draft is a deletion ``` What the library checks for you: the URL is absolute `http(s)`, has no credentials, fragment or control characters, and belongs to a host you hold a key for (`strict_hosts`). What it cannot check: `noindex`, `robots.txt`, a canonical pointing elsewhere, the status code the page answers — that is the rule author's job today, and the job of the `verify` add-on (`check --sample`) in a later release. # Configuration [Русская версия](configuration.ru.md) `IndexNowKit\Config` is an immutable value object shared by every adapter. It is built in one of three ways and validated in the constructor, so a broken setup fails at boot instead of at the first submission. ```php use IndexNowKit\Config; $config = Config::fromArray([...]); // framework config files $config = Config::fromEnv(); // INDEXNOW_* environment variables $config = new Config(key: '...', baseUrl: '...'); // named arguments $config = $config->with(dryRun: true); // immutable copy ``` ## Options `fromArray()` takes the nested shape below; it is the canonical schema every language adapter mirrors. ```php Config::fromArray([ 'enabled' => true, 'key' => $_ENV['INDEXNOW_KEY'], 'hosts' => [ 'www.example.com' => 'KEY-FOR-EXAMPLE', 'shop.example.com' => [ 'key' => 'KEY-FOR-SHOP', 'key_location' => 'https://shop.example.com/keys/indexnow.txt', 'base_url' => 'https://shop.example.com', ], ], 'strict_hosts' => true, 'key_location' => null, 'base_url' => 'https://www.example.com', 'engines' => ['api'], 'dispatch' => 'sync', 'batch' => ['max_urls' => 10000], 'debounce' => ['per_url' => 600], 'throttle' => ['max_requests_per_minute' => 60], 'http' => ['timeout' => 10.0, 'user_agent' => null], 'serve_key_file' => true, 'dry_run' => false, 'environment' => $_ENV['APP_ENV'] ?? null, ]); ``` | Option | Constructor argument | Default | Meaning | |---|---|---|---| | `enabled` | `enabled` | `true` | `false` drops every submission; the URLs come back as `skipped` results with reason `disabled`, logged at `info` | | `key` | `key` | `null` | default key, 8-128 characters of `[A-Za-z0-9-]`, used for every host not listed in `hosts` | | `hosts` | `hosts` | `[]` | `host => key`, or `host => {key, key_location?, base_url?}` | | `strict_hosts` | `strictHosts` | `false` | apply the default key **only** to the `base_url` host; every other host needs a `hosts` entry or its URLs are skipped | | `key_location` | `keyLocation` | `null` | absolute URL of the key file when it is not `https://{host}/{key}.txt` | | `base_url` | `baseUrl` | `null` | absolute site URL; resolves relative URLs and is required outside HTTP requests | | `engines` | `engines` | `['api']` | engine names (`api`, `yandex`, `bing`, `naver`, `seznam`, `yep`, `internetarchive`, `amazon`) or full endpoint URLs | | `dispatch` | `dispatch` | `'sync'` | delivery mode defined by the adapter; the core validates the identifier and reports it | | `batch.max_urls` | `batchMaxUrls` | `10000` | URLs per request; `Config::MAX_BATCH_URLS` is the protocol maximum — a ceiling, not a target: smaller batches are accepted just as well | | `debounce.per_url` | `debouncePerUrl` | `600` | seconds during which the same URL is not re-sent; `0` disables debouncing | | `throttle.max_requests_per_minute` | `throttleMaxRequestsPerMinute` | `60` | outgoing requests per minute, per process; `0` = unlimited | | `http.timeout` | `httpTimeout` | `10.0` | seconds, applied only to clients the library creates itself | | `http.user_agent` | `userAgent` | `null` | overrides `indexnowkit-php/ (+https://github.com/indexnowkit/php)` | | `key_file.enabled` | `serveKeyFile` | `true` | whether an adapter should answer `GET /{key}.txt`; `serve_key_file` is the deprecated name and wins when both are set | | `key_file.cache_max_age` | `keyFileMaxAge` / `keyFileHeaders()` | `300` | `Cache-Control: max-age` of the key file response; short on purpose, a cached old file turns every submission into a 403 after a rotation | | `debounce.store` | `debounceStore` | `null` | `memory` (per process), `none`, or an id the adapter resolves to its shared cache; `null` = the adapter's default (Laravel `cache`, bundle `cache.app`, Yii2 `cache`, plain PHP `memory`) | | `http.client` | `httpClient` | `null` | id or class of a PSR-18 client the adapter resolves; `null` = discovery | | `dry_run` | `dryRun` | `false` | log the request instead of sending it | | `environment` | `environment` | `null` | application environment; drives the non-production safety net below | | `production_environments` | `productionEnvironments` | `['prod', 'production']` | environment names (case-insensitive) that count as production; replaces the default list | | `previous_key` | `previousKey` | `null` | the key before a rotation: still accepted by the key file, never submitted; also `hosts..previous_key` | | `hosts..engines` | `hostEngines` / `endpointsFor()` | inherit `engines` | engines for one host only | | `engine_aliases` | `engineAliases` / `resolveEngine()` | `{}` | short names for custom endpoints, usable wherever an engine is named | | `locale_hosts` | `localeHosts` / `hostForLocale()` | `{}` | locale => host; rules with `locales` and no `host` generate each locale on its host | | `logging.max_body` | `logBody` | `300` | bytes of an engine response body kept in a failure log line | | `max_url_length` | `maxUrlLength` | `2048` | URLs above it are skipped as `invalid_url` | | `debounce.key_prefix` | `debounceKeyPrefix` | `'indexnowkit_'` | cache key prefix of a shared debounce store | | `logging.max_urls` | `logUrls` / `logSample()` | `20` | URLs listed in one log line; `0` = counts only | | `logging.forbidden_escalation` | `forbiddenEscalation` | `5` | consecutive 403s per host before the log escalates to `critical` | | `logging.levels` | `logLevels` / `logLevel()` | `{}` | per-outcome PSR-3 level overrides; events and defaults in `Config::LOG_EVENTS` | | `retry.max_attempts`, `retry.base_delay`, `retry.multiplier`, `retry.max_delay`, `retry.server_error_delay` | `retryPolicy()` | `3`, `60`, `2.0`, `3600`, `5` | the `RetryPolicy` for queue handlers and `RetryingSubmitter` | | `resolver.max_via_depth`, `resolver.max_via_fanout` | `resolverMaxViaDepth`, `resolverMaxViaFanout` | `3`, `100` | limits of `via:` traversal in `AttributeUrlResolver`. `IndexNowKit::create()` does not build that resolver: the adapter that does passes `resolverMaxViaDepth`, `resolverMaxViaFanout` and `localeHosts` to it | | `collector.max_urls` | `collectorMaxUrls` | `0` | `IndexNowKit::collect()` flushes early at this size; `0` = only on `flush()` | | `collector.detect_leaks` | `collectorDetectLeaks` | `true` | shutdown warning about collected, never flushed URLs | | `normalizer.strip_tracking_params` | `normalizerStripTrackingParams` | `true` | drop `utm_*`, `gclid`, `fbclid`, `yclid`, … (`Url\CanonicalUrlNormalizer::TRACKING_PARAMS`, a growing list) from the query before de-duplication, debounce and submission: external traffic sources append them, routing never generates them | | `normalizer.tracking_params` | `normalizerTrackingParams` | `[]` | more query parameters to drop: names (`ref`) or prefixes (`mtm_*`), case-insensitive | | `normalizer.trailing_slash` | `normalizerTrailingSlash` | `'keep'` | `keep` submits the path as generated; `add` ends every path without an extension with `/`; `strip` removes the trailing `/` except on the root. Only when the site has a canonical form: the two forms are different pages otherwise | | `normalizer.sort_query` | `normalizerSortQuery` | `false` | order the query parameters by name (stable), so `?b=1&a=2` and `?a=2&b=1` are one URL | The `normalizer.*` options are applied by `Url\UrlNormalizerFactory::fromConfig()`, which every adapter and `IndexNowKit::create()` use to build the normalizer: `Url\UrlNormalizer` (absolute URL, host, port, dot-segments) wrapped in `Url\CanonicalUrlNormalizer`. Turning `strip_tracking_params` on or off changes the debounce keys of URLs that carried such parameters once. Constants worth referencing instead of hard-coding: `Config::MAX_BATCH_URLS` (10000), `Config::DEFAULT_BATCH_MAX_URLS`, `Config::DEFAULT_DEBOUNCE_PER_URL` (600), `Config::DEFAULT_THROTTLE_PER_MINUTE` (60), `Config::DEFAULT_HTTP_TIMEOUT` (10.0), `Config::PRODUCTION_ENVIRONMENTS` (`['prod', 'production']`), `Config::DEFAULT_MAX_URL_LENGTH`, `Config::DEFAULT_LOG_URLS`, `Config::DEFAULT_FORBIDDEN_ESCALATION`, `Config::DEFAULT_RETRY_*`, `Config::DEFAULT_RESOLVER_MAX_VIA_*`, `Config::LOG_EVENTS`. ## One concept, three keys The adapters share the core keys under the same names and add a few of their own; some concepts have a different key (or a different value set) per framework. The tables below are generated from the code (`bin/config-table`) and checked in CI, so they are the current truth; the prose of each adapter's `docs/configuration.md` explains the semantics. _Generated by `bin/config-table` from `Config::OPTIONS`, `SitemapConfig::OPTIONS`, the bundle configuration tree, `ConfigFactory::LARAVEL_OPTIONS` and `ConfigFactory::YII_OPTIONS`; do not edit by hand._ ### Core keys: the same name in every adapter Every key of `Config::OPTIONS` is accepted under this name by the Symfony bundle (`indexnowkit:`), the Laravel package (`config/indexnow.php`) and the Yii2 component (`options`). The default column is the one the core ships, as the bundle declares it in its configuration tree (`—` = unset); the two exceptions are in the synonyms table: `dispatch` (`auto` in Symfony and Yii2, `queue` in Laravel) and `debounce.store` (`cache.app` / `cache` / `cache`). `environment` comes from `kernel.environment` / `APP_ENV` / `YII_ENV` unless set. | Key | Default | |---|---| | `enabled` | `true` | | `key` | — | | `hosts` | `[]` | | `key_location` | — | | `base_url` | — | | `engines` | `[api]` | | `dispatch` | `auto` | | `serve_key_file` | deprecated alias of `key_file.enabled` | | `dry_run` | `false` | | `strict_hosts` | `false` | | `environment` | — | | `production_environments` | `[prod, production]` | | `max_url_length` | `2048` | | `previous_key` | — | | `key_file.enabled` | `true` | | `key_file.cache_max_age` | `300` | | `batch.max_urls` | `10000` | | `debounce.per_url` | `600` | | `debounce.key_prefix` | `indexnowkit_` | | `debounce.store` | `cache.app` | | `throttle.max_requests_per_minute` | `60` | | `http.timeout` | `10` | | `http.user_agent` | — | | `http.client` | — | | `logging.max_urls` | `20` | | `logging.forbidden_escalation` | `5` | | `logging.levels` | `[]` | | `logging.max_body` | `300` | | `engine_aliases` | `[]` | | `locale_hosts` | `[]` | | `retry.max_attempts` | `3` | | `retry.base_delay` | `60` | | `retry.multiplier` | `2` | | `retry.max_delay` | `3600` | | `retry.server_error_delay` | `5` | | `resolver.max_via_depth` | `3` | | `resolver.max_via_fanout` | `100` | | `collector.max_urls` | `0` | | `collector.detect_leaks` | `true` | | `normalizer.strip_tracking_params` | `true` | | `normalizer.tracking_params` | `[]` | | `normalizer.trailing_slash` | `keep` | | `normalizer.sort_query` | `false` | `hosts` (per-host keys, `hosts..{key, key_location, base_url, engines, previous_key}`) is accepted everywhere too. ### Sitemap keys (`indexnowkit/sitemap`) The `sitemap` block is the same in the three adapters and is owned by the sitemap package: `sitemap.enabled`, `sitemap.url`, `sitemap.max_depth`, `sitemap.max_sitemaps`, `sitemap.max_bytes`, `sitemap.allow_foreign_hosts`, `sitemap.spool`, `sitemap.spool_dir`, `sitemap.fetch_retries`. ### One concept, three keys | Concept | Symfony (`indexnowkit:`) | Laravel (`config/indexnow.php`) | Yii2 (`options`) | Notes | |---|---|---|---|---| | Delivery mode | `dispatch` | `dispatch` | `dispatch` | `auto` (Messenger when a transport is set, else `sync`), `messenger`, `sync`, `none` — Symfony; `queue` (default), `sync`, `none` — Laravel, no `auto`; `auto` (default: `queue` when the queue component exists, else `sync`), `queue`, `sync`, `none` — Yii2 | | Queue / transport | `messenger.transport` | `queue.connection` | `queue.component` | Symfony: a `framework.messenger.transports` name (the bundle routes `SubmitUrlsMessage` to it); Laravel: a `queue.connections` name (default: the app default); Yii2: the yii2-queue component id (default `queue`) | | Queue delay / extras | `messenger.delay` | `queue.delay` | `queue.delay` | Symfony also `messenger.stamps`, `messenger.bus`; Laravel also `queue.queue`; Yii2 also `queue.ttr`, `queue.priority` | | Locales for `locales: all` | `framework.enabled_locales` | `router.locales` | `router.languages` | Symfony reads the framework setting; Laravel and Yii2 list them in the package configuration (`router.locale_parameter` / `router.language_parameter` name the route parameter; `router.set_app_locale` / `router.set_app_language` switch the application locale while generating) | | ORM hook switch | `doctrine.enabled` | `eloquent.enabled` | `active_record.enabled` | Symfony also `doctrine.listener_priority`, `doctrine.connections`; Yii2 also `active_record.models` (classes you cannot annotate) | | Key file route | `key_file.path` | `key_file.path` | `key_file.pattern` | Symfony/Laravel: a path with `{key}` (default `/{key}.txt`); Yii2: a URL rule pattern (default `.txt`); all three: `key_file.enabled`, `key_file.cache_max_age`; Symfony/Laravel also `key_file.host`, `key_file.route_name`; Laravel also `key_file.middleware` | | Log destination | `logging.channel` | `logging.channel` | `logging.category` | Monolog channel (Symfony, default `indexnow`), log channel name (Laravel), Yii log category (default `indexnow`) | | Debounce store | `debounce.store` | `debounce.store` | `debounce.store` | Same key, different values: a PSR-6 pool service id (Symfony, default `cache.app`), a cache store name (Laravel, default `cache` = the default store), a cache component id (Yii2, default `cache`); `memory` and `none` everywhere | | HTTP client | `http.client` | `http.client` | `http.client` | Same key: a service id (PSR-18 or symfony/http-client) in Symfony, a container binding or class in Laravel, a component id or class in Yii2; unset = PSR-18 discovery | ### Adapter-only keys | Adapter | Keys | |---|---| | Symfony | `messenger.bus`, `messenger.transport`, `messenger.delay`, `messenger.stamps`, `key_file.path`, `key_file.host`, `key_file.route_name`, `logging.channel`, `flush.priority`, `flush.console_priority`, `profiler.enabled`, `doctrine.enabled`, `doctrine.listener_priority`, `doctrine.connections` | | Laravel | `queue.connection`, `queue.queue`, `queue.delay`, `key_file.path`, `key_file.host`, `key_file.route_name`, `key_file.middleware`, `router.locales`, `router.locale_parameter`, `router.set_app_locale`, `eloquent.enabled`, `logging.channel` | | Yii2 | `queue.component`, `queue.ttr`, `queue.delay`, `queue.priority`, `key_file.pattern`, `router.languages`, `router.language_parameter`, `router.set_app_language`, `active_record.enabled`, `active_record.models`, `logging.category` | ## Environment variables `Config::fromEnv()` reads `getenv()` merged with `$_SERVER` and `$_ENV`. Pass your own array as the first argument to read from somewhere else, and a second argument to change the `INDEXNOW_` prefix. Empty strings count as unset. | Variable | Option | |---|---| | `INDEXNOW_ENABLED` | `enabled` (any boolean literal `filter_var` accepts) | | `INDEXNOW_KEY` | `key` | | `INDEXNOW_PREVIOUS_KEY` | `previous_key`: the key before a rotation, still served and accepted by the key file, never submitted | | `INDEXNOW_HOSTS` | `hosts`, as `host=key,host2=key2`; per-host `key_location`/`base_url` need `fromArray()` | | `INDEXNOW_STRICT_HOSTS` | `strict_hosts` | | `INDEXNOW_KEY_LOCATION` | `key_location` | | `INDEXNOW_BASE_URL` | `base_url` | | `INDEXNOW_ENGINES` | `engines`, comma-separated (`api` or `yandex,bing`) | | `INDEXNOW_DISPATCH` | `dispatch` | | `INDEXNOW_BATCH_MAX_URLS` | `batch.max_urls` | | `INDEXNOW_DEBOUNCE_PER_URL` | `debounce.per_url` | | `INDEXNOW_THROTTLE_PER_MINUTE` | `throttle.max_requests_per_minute` | | `INDEXNOW_HTTP_TIMEOUT` | `http.timeout` | | `INDEXNOW_USER_AGENT` | `http.user_agent` | | `INDEXNOW_KEY_FILE_ENABLED` (`INDEXNOW_SERVE_KEY_FILE` still wins) | `key_file.enabled` | | `INDEXNOW_KEY_FILE_CACHE_MAX_AGE` | `key_file.cache_max_age` | | `INDEXNOW_DEBOUNCE_STORE` | `debounce.store` | | `INDEXNOW_HTTP_CLIENT` | `http.client` | | `INDEXNOW_DRY_RUN` | `dry_run` | | `INDEXNOW_ENV`, else `APP_ENV` | `environment` | | `INDEXNOW_PRODUCTION_ENVIRONMENTS` | `production_environments`, comma-separated | | `INDEXNOW_MAX_URL_LENGTH` | `max_url_length` | | `INDEXNOW_LOG_URLS`, `INDEXNOW_FORBIDDEN_ESCALATION` | `logging.max_urls`, `logging.forbidden_escalation` | | `INDEXNOW_RETRY_MAX_ATTEMPTS`, `INDEXNOW_RETRY_BASE_DELAY`, `INDEXNOW_RETRY_MULTIPLIER`, `INDEXNOW_RETRY_MAX_DELAY`, `INDEXNOW_RETRY_SERVER_ERROR_DELAY` | `retry.*` | ## Hosts, keys and `strict_hosts` Sub-domains are separate hosts for IndexNow: each needs its own key file. Three layouts: - **One site.** Set `key` and `base_url`. Every host you submit uses that key. - **Several sites, one key each.** Fill `hosts`. Hosts missing from the map still fall back to `key`. - **Several sites, nothing else.** Set `strict_hosts: true`. The default key then applies only to the `base_url` host; URLs of any other unlisted host are skipped with reason `no_key` instead of being announced under someone else's key. Recommended whenever URLs can come from user input or from a multi-tenant database. `hosts..key_location` overrides the key file URL for that host only, and must be on that host. `hosts..base_url` gives the host its own absolute base for URL generation outside a request — a console command or a queue worker has no request context, so without it every site would be generated on the single global `base_url`. `Config::baseUrlFor($host)` returns that per-host base, falling back to `base_url` when the host is the base host, and `null` otherwise. Keys can be enumerated with `Config::$hosts`, `Config::$keyLocations` and `Config::$hostBaseUrls` (all lower-cased host maps). To load keys from a database or a tenant registry, implement `Key\KeyProviderInterface` instead. ## The dry-run safety net `Config::fromArray()` switches `dry_run` on by itself when **all** of these hold: no `key`, no `hosts`, an `environment` is given, and it is not in `production_environments` (default `Config::PRODUCTION_ENVIRONMENTS`). A developer who never sets `INDEXNOW_KEY` locally therefore gets logging instead of a boot failure, and never reaches the real API. The reverse case is worth alerting on: `dry_run` on while `environment` says production means nothing is being submitted at all. `Config::isProduction()` reports it, and `Check\Checker` raises it as an **error** rather than a warning in that combination. ## Validation The constructor throws `Exception\ConfigurationException` for: - `enabled` without `key`, `hosts` or `dry_run`; - a `key` (or any host key) outside `[A-Za-z0-9-]{8,128}`; - a `hosts` key that is not a bare host name (scheme, port or path present); - `base_url` that is not an absolute `http(s)` URL, or carries credentials; - `key_location` that is not an absolute `http(s)` URL with a path, or is not on the `base_url` host — engines only accept a key file served from the submitted host; - `hosts..key_location` or `hosts..base_url` pointing at a different host; - `batch.max_urls` outside `1..10000`, negative `debounce.per_url` or `throttle.max_requests_per_minute`, `http.timeout` at or below zero, an empty `engines` list; - a `dispatch` value that is not a short identifier, a `http.user_agent` containing line breaks; - `strict_hosts` without any known host; - an engine name that is neither a known engine nor an `https` endpoint (plain `http` is allowed only on loopback hosts, for mock servers). `Config::fromArray()` additionally rejects non-numeric values for numeric options rather than silently falling back to the default. ## Deriving configurations `with()` takes constructor argument names and returns a validated copy; an unknown name throws. ```php $probe = $config->with(dryRun: false, engines: ['yandex']); $config->withDryRun(true); // shorthand $config->userAgent(); // the effective User-Agent string $config->baseHost(); // lower-cased host of base_url, or null ``` ## Detecting typos in adapter config `Config::OPTIONS` lists every key `fromArray()` understands, in dotted form. `Config::unknownOptions($data, $allowed)` returns the keys of an array that are neither core options nor listed in `$allowed`, so an adapter can warn about `debounce.per_urls` instead of silently ignoring it. List nested keys as `block.key`, never as a bare `block`: a bare name stops the check from looking inside the block. Adapters get this through `Adapter\ConfigFactory::load()` (`ownedOptions:`), which also merges the adapter's defaults, resolves `dispatch: auto` and turns an invalid value into a `critical` log line and a disabled `Config` instead of an exception. ```php $unknown = Config::unknownOptions($userConfig, ['messenger', 'messenger.bus', 'doctrine.enabled']); if ($unknown !== []) { $logger->warning('indexnow: unknown option(s): {options}', ['options' => implode(', ', $unknown)]); } ``` Nested arrays are checked one level deep by dotted path; `hosts` is always accepted because its keys are host names. Naming a block in `$allowed` (for example `messenger`) allows the whole block, so an adapter lists either the block name or the individual dotted paths it owns. # Operations Everything here is about the question an operator actually asks: *my page changed, why was nothing submitted?* — and, before that, about not shipping a setup that submits the wrong thing. ## Production checklist Before the first real submission, and again after every deployment that touches the configuration: 1. **Key and base URL.** `INDEXNOW_KEY` (8–128 characters of `[A-Za-z0-9-]`) and `base_url` are set; every host you submit serves `https:///.txt` with `200`, `text/plain`, the key as the body and no redirect. 2. **`check --strict` is green** in the environment that submits (`bin/console indexnow:check --strict`, `php artisan indexnow:check --strict`, `php yii indexnow/check --strict`): exit code 0. Put it in the deploy pipeline; it exits 1 on any error and, with `--strict`, on any warning. `check --json` (schema `docs/check.schema.json` of `indexnowkit/console`, codes in [check-codes.md](check-codes.md)) is the form for monitoring: alert on `status` and on the codes, never on the texts. `config --json` is what to paste into a bug report. 3. **`strict_hosts: true`** whenever a `hosts` map exists or the application answers under more than one hostname (a staging copy, an internal name, the apex next to `www`). 4. **A shared debounce store.** `debounce.store` is a cache that web requests and workers share, not `memory`. 5. **The queue is monitored.** `dispatch: queue` / `messenger` runs a worker; failed jobs are visible; the 403 "rejected permanently" line has an owner. 6. **Staging cannot submit.** Outside production set `INDEXNOW_DRY_RUN=1` (or `INDEXNOW_ENABLED=0`) and `key_file.enabled: false`, so the staging host neither sends nor serves the production key. Since core 0.6, `check` fails on a staging copy that has a key and no `dry_run` setting; a preview environment that submits on purpose says `dry_run: false` explicitly. 7. **Alerts on three lines**: the 403 escalation (`critical`), `invalid configuration, IndexNow is disabled` (`critical`), and `collected URL(s) discarded` (`warning`). The monitoring rules below say how. 8. **Short key-file caching.** `key_file.cache_max_age` ≤ 300 and the CDN honours it: after a rotation the old file must not be served for a day. 9. **`previous_key` removed** once every engine answers 200 for the new key (`check --live`). 10. **Someone looks at the result**: Bing Webmaster Tools → IndexNow Insights, Yandex.Webmaster → Indexing → Reindex pages. IndexNow is a notification; the share of submitted URLs that are in the index after a few days is the number that says whether the setup works. ## What IndexNow is, and is not A submission tells an engine that a URL changed. Whether and when the page is crawled and indexed is the engine's decision; a `200` from the endpoint means "received", nothing more. Google does not participate. The Bing URL Submission API and Google's Indexing API are different protocols with their own quotas and are not covered by this library. Where to see the result: Bing Webmaster Tools (IndexNow Insights: received URLs, crawl outcome, errors per key) and Yandex.Webmaster (Indexing → Reindex pages, and the crawl statistics). A useful success metric is the share of submitted URLs present in the index after a few days, and the time between a change and the updated snippet. ## Deleted pages: what your site must return An engine that receives a URL fetches it. The response decides what happens to the page in the index: | Situation | Return | Effect | |---|---|---| | Gone for good | `410 Gone` | the fastest removal; `404` works too but is treated as "maybe temporary" | | Temporarily unavailable | `404` (or `503` with `Retry-After` for maintenance) | the page stays indexed for a while | | Moved | `301` to the new URL, and submit **both** URLs (the old one is resolved as a deletion, the new one as an update — the ORM adapters do this on a slug change) | the index follows the redirect | | A "not found" page that answers `200` (soft 404) | do not: fix it to `404`/`410` | the engine keeps a useless page and trusts the site less | | Redirect to the home page | do not: `410` or `301` to the closest equivalent | same as a soft 404 | The library sends the URL of a deleted object exactly once; the site's answer does the rest. ## What not to submit The engines fetch what you submit, and a URL that is not meant to be indexed costs trust and quota: - pages with `` or an `X-Robots-Tag: noindex` header; - paths that `robots.txt` disallows (the engine cannot fetch them; some count it as an error against the key); - non-canonical URLs: tracking parameters, sort/filter variants, session ids, `http://` next to `https://`, the apex next to `www` — submit the `` target only; - URLs that answer `3xx`, `4xx` or `5xx` (except the deletions above); - drafts, previews, unpublished or access-restricted pages. What protects you today: the URL normalizer accepts only absolute `http(s)` URLs, strips fragments and default ports, and rejects URLs with credentials or control characters; `strict_hosts` keeps foreign hosts out; the `when` guard of a rule keeps drafts out (`when: 'isPublished'`), and a `published → draft` change is submitted as a deletion. What it cannot see: a `noindex` tag, a `robots.txt` rule, a canonical pointing elsewhere. Those are the job of the rule (do not declare a rule on such a model, or guard it with `when`) — and of the `verify` add-on that a later release adds (a pre-flight fetch of a sample of URLs by `check --sample`). ## Log channel and levels Every message starts with `indexnow: ` and goes to the PSR-3 logger you inject. Framework adapters put it on a dedicated channel — `indexnow` in the Symfony bundle — so `tail -f var/log/prod.indexnow.log` shows the whole story. ### Delivery outcomes (`Client`) | Level | Message | |---|---| | `debug` | `indexnow: {engine} accepted {count} URL(s) for {host}` | | `info` | `indexnow: {engine} accepted {count} URL(s) for {host}, key verification pending (202)` | | `info` | `indexnow: dry-run POST {endpoint} {body}` | | `warning` | `indexnow: skipping {count} URL(s) for unmanaged host {host}: no key configured (add it to "hosts" or set base_url)` | | `warning` | `indexnow: {engine} could not process URLs for {host} (422): URLs do not belong to the host or keyLocation is invalid` | | `warning` | `indexnow: {engine} rate limited (429) for {host}, retry after {retry_after}s` | | `warning` | `indexnow: {engine} server error {status} for {host}` | | `warning` | `indexnow: {engine} transport error for {host}: {error}` | | `error` | `indexnow: {engine} rejected the key for {host} (403). Check that https://{host}/{key}.txt is reachable and contains the key (run the check command of your adapter, e.g. indexnow:check).` | | `error` | `indexnow: {engine} rejected the request as malformed (400): {body}` | | `error` | `indexnow: {engine} unexpected status {status} for {host}: {body}` | | `error` | `indexnow: {engine} HTTP client failure for {host}: {error}` | | `error` | `indexnow: cannot encode {count} URL(s) for {host} as JSON: {error}` | | `error` | `indexnow: throttle failed, sending without rate limiting: {error}` | | `critical` | the 403 message plus `{consecutive} consecutive failures: submissions for this host are not being indexed.` | The 403 escalation is the one line to page on. `logging.forbidden_escalation` is 5 by default: the fifth consecutive 403 for a host is logged once at `critical`, further ones drop back to `warning` so they do not spam, and any non-403 response resets the counter. Since core 0.8 the counter lives in the cache behind `debounce.store` (the adapters pass it to `Client` as the PSR-16 "failure cache"; plain PHP: `IndexNowKit::create(..., failureCache: $cache)`), so PHP-FPM workers and queue workers count together and the fleet writes the `critical` line once per streak: the keys are `403.` and `…_escalated`, kept for an hour after the last 403. With `debounce.store: memory` or `none` the counter stays in the process, where every worker counts its own 403s and pages on its own fifth failure — alert on the `warning` rate of `reason=invalid_key` as well there. A cache that throws is logged once (`failure cache unavailable, counting 403s per process`) and the process counts on. Every other level in these tables is the default of `logging.levels` (`Config::LOG_EVENTS`) and can be raised or lowered per outcome; `logging.max_urls` decides how many URLs a line lists (0 for PII-sensitive logs). Keys are masked everywhere, including inside response bodies and exception messages. ### Configuration (`Adapter\ConfigFactory`, adapters) | Level | Message | |---|---| | `warning` | `indexnow: unknown option(s) in the indexnow configuration: {options}` (dotted keys, the typo check) | | `critical` | `indexnow: invalid configuration, IndexNow is disabled until it is fixed: {error} (run "{check}")` — nothing is sent until the value is fixed | ### Submission pipeline (`Submitter`) | Level | Message | |---|---| | `info` | `indexnow: disabled (enabled: false), dropping {count} URL(s)` | | `warning` | `indexnow: dropping URL: {error}` | | `warning` | `indexnow: debounce store unavailable, submitting without de-duplication: {error}` | | `warning` | `indexnow: debounce store failed after a successful submission, URLs may be re-sent within {ttl}s: {error}` | | `debug` | `indexnow: debounced {count} URL(s) submitted within the last {ttl}s` | | `error` | `indexnow: result listener {listener} failed: {error}` / `indexnow: result event listener failed: {error}` | `disabled` is at `info` on purpose: it is the most common "nothing is happening at all" state, and `debug` is filtered out in most production setups. ### Resolution (`GuardedUrlResolver`, `ObjectChangeHandler`) | Level | Message | |---|---| | `debug` | ``indexnow: {class} rule "{rule}" skipped for {event}: `when` is false`` | | `debug` | ``indexnow: {class} rule "{rule}" ignores this update (fields {changed} vs filter {fields}, or `when` unchanged and false)`` | | `debug` | ``indexnow: no URLs for {class} ({event}): no rule applies (no #[IndexNow], event not subscribed, or `when` is false)`` | | `debug` | `indexnow: {class} does not subscribe to {event}` | | `warning` | `indexnow: #[IndexNow(via: "{via}")] on {class} stops after {max} related objects` | | `error` | `indexnow: invalid #[IndexNow] on {class}: {error}` | | `error` | ``indexnow: cannot evaluate `when` of {class} rule "{rule}": {error}`` | | `error` | `indexnow: cannot classify the change of {class} for rule "{rule}": {error}` | | `error` | `indexnow: cannot resolve URLs for {class} rule "{rule}" ({event}): {error}` | Turn the `indexnow` channel to `debug` while diagnosing: the four debug lines above are the difference between "nothing happened" and "the rule decided not to". ### ORM hooks (`Hook\ObserverHelper`, the observers of every adapter) | Level | Message | |---|---| | `debug` | `indexnow: {source} ({event}) -> {url}` — one line per resolved URL, with the rule that produced it | | `error` | `indexnow: cannot resolve the URLs of {class}: {error}` — the hook went on, the object was not submitted | | `error` | `indexnow: cannot collect {count} URL(s): {error}` | ### Queue workers (`Retry\WorkerOutcome`, the jobs of every adapter) | Level | Message | |---|---| | `info` | `indexnow: {count} URL(s) of job {id} will be retried{delay}{attempt}` — `{delay}` is ` in {n}s` where the job sets the delay (Laravel), `{attempt}` is ` (attempt {n})` where the job knows it | | `error` | `indexnow: giving up on {count} URL(s) of job {id} after {attempt} attempt(s)` (Laravel and yii2-queue; Messenger reports exhausted retries itself) | | `error` | `indexnow: {count} URL(s) of job {id} rejected permanently ({reasons}); run "{check}"` — `{reasons}` lists ` `: `api 403`, `yandex 422` | ### Delivery hand-off | Level | Message | |---|---| | `warning` | `indexnow: {count} collected URL(s) discarded: the unit of work ended without flush() (request end hook not run?)` | | `debug` | `indexnow: discarding {count} staged URL(s), transaction rolled back` / `..., savepoint rolled back` | | `debug` | `indexnow: throttle limit of {per_minute} requests/min reached, waiting {wait_ms} ms` | | `error` | `indexnow: sync dispatch of {count} URL(s) failed, they are lost: {error}` / `indexnow: dispatch of {count} URL(s) failed, they are lost: {error}` | ## Metrics `Result::metricLabels()` returns low-cardinality labels ready for a counter: `status`, `engine`, `reason`, `http_code`, `retryable`. The host is deliberately absent because it is unbounded in multi-tenant setups; add `$result->host` yourself if your cardinality budget allows. ```php $indexNow->submitter->addListener(function (IndexNowKit\Result $result) use ($metrics): void { $metrics->counter('indexnow_results_total', $result->metricLabels())->inc(); $metrics->counter('indexnow_urls_total', $result->metricLabels())->incBy($result->urlCount()); }); ``` A listener that throws is logged and ignored; delivery is never affected. A decorator around `SubmitterInterface` must forward `addListener()`, or listeners registered on the outer object never fire. Alert on: `reason=invalid_key` (the key file broke), a sustained `reason=rate_limited`, `status=failed` with `retryable=false`, and the collector-discard warning above. ## Monitoring rules Four rules cover what goes wrong in production; the first two page, the other two open a ticket. | # | Signal | Threshold | Meaning and action | |---|---|---|---| | 1 | `critical` on the `indexnow` channel | any | the key file broke (403 ×5) or the configuration is invalid and IndexNow is off: run `check`, fix, redeploy | | 2 | results with `status=failed`, `retryable=false` (403, 422, 400) | > 0 in 15 min | permanent rejections: the key file, URLs of a foreign host, or a bug — `explain` one of the URLs | | 3 | results with `reason=rate_limited` | sustained for 10 min | the engine throttles you: lower `throttle.max_requests_per_minute`, raise `batch.max_urls` usage, or wait; retries follow `Retry-After` | | 4 | `warning: … collected URL(s) discarded` | any | a request or job ended without `flush()`: the runtime skipped the terminate hook (early `exit()`, fatal error, long-running runtime) — prefer a queued dispatch there | Everything else the library logs at `warning` is per request and self-healing (a cache blip, a 5xx that the queue retries): count it, do not page on it. A `debug`-level channel in production is fine volume-wise only with `logging.max_urls: 0`. **Sentry filter.** The library logs at `warning` for outcomes the queue retries; forwarding every one of them to Sentry turns a rate-limited hour into hundreds of events. Keep `error` and above from the `indexnow` channel, drop the rest: ```php // sentry.php / config/sentry.php — keep errors, drop the per-request warnings of the library 'before_send' => static function (\Sentry\Event $event): ?\Sentry\Event { $level = (string) $event->getLevel(); if ($event->getLogger() === 'indexnow' && !\in_array($level, ['error', 'fatal'], true)) { return null; } return $event; }, ``` (Symfony: the channel name is `logging.channel`, default `indexnow`; Laravel: the log channel of `indexnow.logging.channel`; Yii2: the `indexnow` category — Yii's Sentry targets pass it as the logger.) ## "My URL was not submitted" Walk it in this order. Each step names the reason or log line that proves it. 1. **Is IndexNow on?** `enabled: false` yields `skipped` / `disabled` and one `info` line per call. 2. **Is it dry-run?** `dry_run` yields `skipped` / `dry_run` and an `info` line with the full body. Outside production a missing key turns this on automatically — that is the intended dev behaviour and a bug in prod. `Checker` reports it as an error when `environment` says production. 3. **Did the rule fire at all?** With an ORM, the `debug` lines above say whether a rule was skipped by `when`, by `events`, or by `fields`. No lines at all means no rules were found: check that the class really carries `#[IndexNow]` and that nothing logged `invalid #[IndexNow] on {class}`. 4. **Did the URL survive normalization?** `warning: indexnow: dropping URL` and `skipped` / `invalid_url`. The usual cause is a relative URL with no `base_url`, in a console command or a worker. 5. **Is there a key for that host?** `warning: skipping ... unmanaged host` and `skipped` / `no_key`. With `strict_hosts` this fires for every host outside `base_url` and the `hosts` map. 6. **Was it debounced?** `skipped` / `debounced`. The same URL is not re-sent within `debounce.per_url`. The debug line reports the count. 7. **Did the engine reject it?** `failed` with reason `invalid_key` (403, key file), `unprocessable` (422, URLs on another host or a bad `keyLocation`), `invalid_request` (400, please report), `rate_limited` or `server_error`. 8. **Did anything get collected but never flushed?** See the next section. ## The collector and units of work `Collector` buffers normalized URLs and is drained once by `IndexNowKit::flush()`. Nothing sends until then. `Collector::reset()` empties the buffer **without delivering**, for long-running runtimes that recycle services between requests. It logs at `warning` when the buffer was not empty. That line means a unit of work ended without a flush and those URLs are gone; it is nearly always the smoking gun for "the entity saved and nothing arrived". Under Symfony, `flush()` runs on `kernel.terminate`, `console.terminate` and `WorkerMessageHandledEvent`. `kernel.terminate` fires only when the SAPI lets it: an early `exit()`, a fatal error before termination, or a reverse-proxy setup that never releases the request can skip it. Under Swoole, RoadRunner or FrankenPHP the behaviour depends on the runtime bridge. In those environments prefer a queue-backed dispatch, where the batch is durably enqueued before the worker moves on, and treat the collector-discard warning as a monitored signal. Long-running custom commands should call `flush()` periodically instead of accumulating for the life of the process. ## Debounce and cache outages The debounce store fails **open**. If `filterRecent()` throws, the submission proceeds without deduplication and logs `debounce store unavailable`; if `markSubmitted()` throws afterwards, the window is not recorded and the URLs may be re-sent within the TTL. Both are warnings, one per `submit()` call, so the noise is bounded by request volume rather than URL volume. The visible symptom of a Redis blip is therefore a burst of duplicate submissions, not lost ones. That is the right trade: a missed submission leaves stale content in the index, a duplicate costs one request. `MemoryDebounceStore` is per process and bounded to 50 000 entries. It is right for CLI runs, tests and single workers; a web application should use `Psr16DebounceStore` on a shared cache so the window survives across processes. ## Throttling in web requests versus workers `TokenBucket` blocks with `usleep()` and counts one token per outgoing HTTP request, per process. Inside a web request it only engages when a single request produces more batches than the limit, so keep `throttle.max_requests_per_minute` comfortably above that, or install `NullThrottle` there and rate-limit in the worker. A throttle that throws never blocks delivery: the request goes out unlimited and an `error` is logged. ## Key rotation Rotating a key breaks submissions until the new key file is reachable, because engines answer 403 for a key whose file they cannot verify. 1. Serve the **new** key file first, alongside the old one if your setup allows it. With the shipped key file route, `previous_key` (`INDEXNOW_PREVIOUS_KEY`) does exactly that: the route answers for both keys, submissions use the new one only. 2. Keep `Cache-Control` short. `KeyFileResponder::DEFAULT_MAX_AGE` is 300 seconds for exactly this reason: a CDN holding the old file for a day means a day of 403s. 3. Switch the configured key. `key:generate --write-env --force` does the whole step in the env file: the new key goes to `INDEXNOW_KEY`, the old one to `INDEXNOW_PREVIOUS_KEY`. It refuses to rotate while `INDEXNOW_PREVIOUS_KEY` still holds the key of an earlier rotation (engines may still verify against it): remove the variable first, or pass `--no-previous` to drop the old key on purpose, or `--yes` to overwrite it. 4. Run the check command. `Checker` fetches every key file over HTTP and compares the body, its `Content-Type` and its `Cache-Control`/`Age` against `key_file.cache_max_age`, and `robots.txt`; `--live` sends a real probe to every endpoint even when `dry_run` is on. With `previous_key` set, the old key file is fetched too: `previous key file OK … rotation window open` (`key_file.previous`) means both keys are served; a warning means the old file is already gone while engines may still verify against it. 5. Watch for the 403 escalation. Five consecutive failures for a host means nothing is being indexed. 6. Remove `previous_key` once `check --live` is green for every host: the line goes away with it. If the key file cannot live at `/{key}.txt`, set `key_location` to its absolute URL on the same host. A `key_location` on a different host is rejected at configuration time, because engines answer 422 for it. The key travels in the JSON body of every submission and in the key file, nowhere else: the library never uses the GET form of the protocol (`?url=…&key=…`), so the key does not end up in access logs, proxy logs or referrers. Logs and exception messages of the library mask it to four characters. # Retries, queues and bulk submissions The core never retries inside a web request. `submit()` returns one `Result` per endpoint × host × batch, and the ones worth trying again carry `retryable: true` (429, 5xx, network failures and unexpected client errors). What you do with them is a deployment decision, not a library one. ## RetryPolicy `Retry\RetryPolicy` decides how long to wait, identically in every adapter. ```php use IndexNowKit\Retry\RetryPolicy; $policy = new RetryPolicy( maxAttempts: 3, // total attempts including the first baseDelay: 60, // seconds before the second attempt after a 429 without Retry-After multiplier: 2.0, maxDelay: 3600, serverErrorDelay: 5, // seconds before the second attempt after 5xx or a network failure ); $delay = $policy->delayAfter($results, $attempt); // null = stop ``` `delayAfter()` returns `null` when the attempt number has reached `maxAttempts` or nothing in the batch is retryable. Otherwise it honours the largest `Retry-After` any result reported, and falls back to `base × multiplier^(attempt-1)`, clamped to `maxDelay`. The base is 60 seconds after a 429, because the engine explicitly asked you to slow down, and 5 seconds after a 5xx or a network blip, which is usually transient. ## In-process retries `Retry\RetryingSubmitter` decorates any `SubmitterInterface` and re-submits the retryable URLs in place. The delay is a blocking `sleep()`, so this belongs in CLI commands, cron jobs and queue workers, never in a web request. ```php use IndexNowKit\Retry\{RetryPolicy, RetryingSubmitter}; $submitter = new RetryingSubmitter($indexNow->submitter, new RetryPolicy(maxAttempts: 3)); $results = $submitter->submit($urls); ``` The returned list holds the last outcome for each URL: results that were retried replace their earlier failure, and results that were never retryable are carried through unchanged. Pass a `$sleeper` callable as the fourth argument to make the retries instant in tests. `RetryingSubmitter` forwards `addListener()` to the inner submitter, so profilers and metrics keep working. Any decorator you write must do the same, or every listener registered on the outer object is silently dropped. ## Queue workers Enqueue the URL list, submit in the worker, re-enqueue what came back retryable. ```php // producer $indexNow->collect($urls); // during the unit of work $indexNow->flush(); // hands the batch to the DispatcherInterface // dispatcher, enqueuing instead of sending use IndexNowKit\Dispatch\CallableDispatcher; $dispatcher = new CallableDispatcher(fn (array $urls) => $queue->push(new SubmitUrls($urls, attempt: 1)), $logger); // worker $results = $indexNow->submit($message->urls); $retry = IndexNowKit\Result::retryableUrls($results); $delay = (new RetryPolicy())->delayAfter($results, $message->attempt); if ($retry !== [] && $delay !== null) { $queue->later($delay, new SubmitUrls($retry, attempt: $message->attempt + 1)); } ``` `Result::retryableUrls()` deduplicates and keeps first-occurrence order. `Result::allUrls()` and `Result::urlsWhere($results, $predicate)` cover the other selections (`Result::urlsOf()`, deprecated since 0.2.0, is gone in 0.4). A worker has no request context, so `base_url` must be configured or every relative URL is dropped as invalid. A dispatcher must never throw into user code: `SyncDispatcher` and `CallableDispatcher` log and swallow. ## Which failures are worth retrying | Outcome | Retry | Why | |---|---|---| | 429 `rate_limited` | yes, after `Retry-After` | the engine will accept it later | | 5xx `server_error` | yes | transient on the engine's side | | network / timeout `transport` | yes | transient on yours | | `unexpected` | check `retryable` | an ill-behaved HTTP client is retryable; a status no engine should return is not | | 403 `invalid_key` | **no** | the key file is wrong; retrying changes nothing, fix it and resubmit | | 422 `unprocessable` | **no** | the URLs do not belong to the host, or `keyLocation` is invalid | | 400 `invalid_request` | **no** | a bug in the library; please report it | | `skipped` (any reason) | **no** | nothing was sent on purpose | ## Bulk imports and migrations A migration that touches 50 000 rows is the one case where the defaults are wrong. - **Do not call `submit()` for 50 000 URLs inside a web request.** Chunk into `Config::MAX_BATCH_URLS`-sized submissions from a CLI command or a worker, with a `RetryingSubmitter` around them. - **Prefer the site's own URL list.** The add-on package in the README family table streams it and filters by modification date, so re-announcing yesterday's changes is one command, not a script. - **Watch the debounce store.** `MemoryDebounceStore` is bounded to 50 000 entries and evicts expired entries first, then the oldest. A run larger than that which also re-touches earlier URLs silently gets a shorter effective debounce window. Use `Psr16DebounceStore` on a shared cache for long runs. - **Throttle in the worker, not in the request.** `TokenBucket` blocks with `usleep()` and counts per process. In a web request keep `throttle.max_requests_per_minute` well above the number of batches one request can produce, or use `NullThrottle` there and rate-limit in the queue instead. - **Rule fan-out is smaller than it looks.** Four rules on a class plus `via: 'category'` means one imported row touches six URLs, but the collector deduplicates within the unit of work and `debounce.per_url` deduplicates across them: a homepage rule costs one submission per debounce window, not one per row. ## Collecting and flushing `Collector` is the per-unit-of-work buffer: `add()`, `all()`, `count()`, `drain()`, `reset()`. `IndexNowKit::flush()` drains it into the dispatcher and does nothing when it is empty. Call it once at the end of the HTTP request, the console command or the queue message. `reset()` empties the buffer **without** delivering, for long-running runtimes that recycle services between requests. It logs a warning when the buffer was not empty, because that means a unit of work ended without a flush and the URLs are gone. Alert on that line. Replace `CollectorInterface` when you need a durable outbox instead. # Testing `IndexNowKit\Testing` is part of the published package, not a dev-only helper: application and adapter test suites are expected to use it. Four doubles, no framework, no HTTP. | Double | Replaces | Gives you | |---|---|---| | `FakeTransport` | `Http\TransportInterface` | recorded POSTs with the decoded body, queued responses and failures | | `ArrayLogger` | `Psr\Log\LoggerInterface` | every record, plus `messages()` with the context interpolated | | `FrozenClock` | `Psr\Clock\ClockInterface` | a clock that only moves when you call `advance()` | | `RecordingDispatcher` | `Dispatch\DispatcherInterface` | the batches handed over, without sending them | ## Asserting what would be submitted ```php use IndexNowKit\{Config, IndexNowKit}; use IndexNowKit\Debounce\NullDebounceStore; use IndexNowKit\Testing\{ArrayLogger, FakeTransport}; $transport = new FakeTransport(); $logger = new ArrayLogger(); $indexNow = IndexNowKit::create( new Config(key: 'test-key-1234', baseUrl: 'https://www.example.com'), transport: $transport, logger: $logger, debounce: new NullDebounceStore(), ); $results = $indexNow->submit(['/posts/hello', '/posts/hello', '/about']); self::assertCount(1, $transport->posts); self::assertSame('https://api.indexnow.org/indexnow', $transport->posts[0]['url']); self::assertSame( ['https://www.example.com/posts/hello', 'https://www.example.com/about'], $transport->posts[0]['body']['urlList'], ); self::assertTrue($results[0]->isSuccess()); ``` Every entry of `$transport->posts` is `['url' => ..., 'json' => ..., 'headers' => ..., 'body' => ...]`, where `body` is the decoded payload, so you assert on `host`, `key`, `keyLocation` and `urlList` directly. `NullDebounceStore` keeps a test from depending on the debounce window. Use `MemoryDebounceStore` with a `FrozenClock` instead when the window is what you are testing. ## Entities and rules ```php $urls = $indexNow->urlsFor($post, IndexNowKit\Event::Updated); self::assertSame(['https://www.example.com/posts/hello'], $urls); foreach ($indexNow->explain($post, IndexNowKit\Event::Updated) as $resolved) { // $resolved->rule, ->class, ->event, ->locale, ->url, ->source() } ``` `urlsFor()` and `explain()` never throw, so a test that expects a broken attribute to be reported asserts on the log instead: ```php self::assertStringContainsString( 'invalid #[IndexNow] on ' . Broken::class, implode("\n", $logger->messages('error')), ); ``` ## Engine responses and failures `willRespond()` queues responses in order; anything beyond the queue gets the constructor default. Queue a `Throwable` to simulate a network failure. ```php use IndexNowKit\Http\Response; use IndexNowKit\Testing\FakeTransport; $transport = (new FakeTransport())->willRespond( new Response(429, '', 30), // rate limited, Retry-After: 30 new Response(200), ); $results = $indexNow->submit(['/a']); self::assertTrue($results[0]->retryable); self::assertSame(30, $results[0]->retryAfter); self::assertSame(IndexNowKit\Reason::RateLimited, $results[0]->reason); $transport->willRespond(FakeTransport::failing('connection refused')); // TransportException on the next POST ``` `FakeTransport::failing()` returns a ready-made `TransportException`; `Response::parseRetryAfter()` is what a real transport uses to turn the header into seconds, and takes a `$now` argument so HTTP-date values are testable. ## Retries without waiting `RetryingSubmitter` takes a sleeper, so a retry test runs instantly and can assert on the delay. Continuing the queue above (429 with `Retry-After: 30`, then 200): ```php use IndexNowKit\Retry\{RetryPolicy, RetryingSubmitter}; $slept = []; $submitter = new RetryingSubmitter( $indexNow->submitter, new RetryPolicy(maxAttempts: 3, baseDelay: 60), $logger, static function (int $seconds) use (&$slept): void { $slept[] = $seconds; }, ); $submitter->submit(['/a']); self::assertSame([30], $slept); // Retry-After won over the exponential base ``` ## Debounce windows ```php use IndexNowKit\Debounce\MemoryDebounceStore; use IndexNowKit\Testing\FrozenClock; $clock = new FrozenClock('2026-01-01 00:00:00'); $indexNow = IndexNowKit::create($config, transport: $transport, debounce: new MemoryDebounceStore($clock)); $indexNow->submit(['/a']); $indexNow->submit(['/a']); self::assertCount(1, $transport->posts); // second call debounced $clock->advance(601); $indexNow->submit(['/a']); self::assertCount(2, $transport->posts); ``` `TokenBucket` takes the same clock plus its own sleeper, so throttling is testable the same way. ## Collecting without sending ```php use IndexNowKit\Testing\RecordingDispatcher; $dispatcher = new RecordingDispatcher(); $indexNow = IndexNowKit::create($config, transport: $transport, dispatcher: $dispatcher); $indexNow->collect(['/a', '/b']); self::assertSame(2, $indexNow->collector->count()); $indexNow->flush(); self::assertSame(['https://www.example.com/a', 'https://www.example.com/b'], $dispatcher->urls()); self::assertCount(1, $dispatcher->batches); self::assertTrue($indexNow->collector->isEmpty()); ``` This is the right double for adapter tests: it proves the unit-of-work hook fired without involving HTTP at all. ## The key file ```php $transport->onGet('https://www.example.com/test-key-1234.txt', new Response(200, 'test-key-1234')); $report = (new IndexNowKit\Check\Checker($config, $indexNow->keys, $transport))->run(); self::assertFalse($report->hasErrors()); ``` Unregistered GET URLs answer `404`, which is what a "key file missing" test wants. ## Dry run `dry_run` exercises the whole pipeline — normalization, deduplication, grouping, key lookup — and stops before the POST. Results come back as `skipped` with reason `dry_run`, and the body is in the `info` log line. ```php $indexNow = IndexNowKit::create($config->with(dryRun: true), transport: $transport); self::assertSame([], $transport->posts); ``` Prefer it in application test suites where you care that a change *would* have been announced; prefer `FakeTransport` where you care about the exact payload. ## Assertions for an adapter's HTTP and command tests The conformance scenarios H01–H05 are the same in every framework, only the way a response or a command output is captured differs. Two static helpers of [`indexnowkit/testing`](../testing/index.md) (`composer require --dev indexnowkit/testing`) hold the assertions, so an adapter test parses its framework's objects and asserts once: ```php use IndexNowKit\Testing\Conformance\CheckOutputAssertions; use IndexNowKit\Testing\Conformance\KeyFileAssertions; // H01: 200, text/plain, the key as the body, Cache-Control with public and max-age, Vary: Host only with a hosts map KeyFileAssertions::assertKeyFileResponse($response->getStatusCode(), $response->headers->all(), $response->getContent(), $key, maxAge: 300, expectVaryHost: true); // H02/H03: an unknown key, another host's key, key_file.enabled: false KeyFileAssertions::assertNotServed($response->getStatusCode()); // H04/H05: the check command CheckOutputAssertions::assertExitCode(0, $exitCode, $output); // the output is the failure message CheckOutputAssertions::assertReady($output, 'www.example.com'); // ": key file OK" and the closing line CheckOutputAssertions::assertKeyFileHint($output, 403); // the status and the hint about what the engines do ``` `Cache-Control` is compared by directive (frameworks order them differently), header names in any case, values as a string or a list. ## Conformance kits for adapters Two abstract PHPUnit cases of `indexnowkit/testing` turn docs/spec/03 into runnable scenarios against *your* wiring (the package is `require-dev`; the core itself ships no PHPUnit code): - `Testing\Conformance\CoreConformanceTestCase` (C01, C03, C04, C06, C09–C12, C14, C19, C20): return the facade your container built and the `FakeTransport` it is wired to; optionally a second configured host for C04. - `Testing\Conformance\OrmConformanceTestCase` (A01–A21, plus A05b/A05c): implement the driver — the transaction verbs of your data layer (`begin()`, `commit()`, `rollback()`), the end of a unit of work (`flush()`, `collectedCount()`), and fixtures with fixed rule shapes (`createPost()`, `createMultiPost()`, `createCategorizedPost()`, `createTag()`, `attachTag()`, `bulkUpdateTitle()`, …). The docblock of the class lists the rules every fixture must carry; the URL conventions (`postUrl()`, `ampUrl()`, `categoryUrl()`, `homeUrl()`) are overridable. `indexnowkit/doctrine` (`tests/OrmConformanceTest.php`) and `indexnowkit/laravel` (`tests/Conformance/`) are the reference drivers. A scenario that does not apply to your framework is documented in your README, not skipped silently. ## Notes for adapter authors - Assert on rules and events through `ObjectChangeHandler::createdEvents()`, `updatedEvents()` and `deletedEvents()` before resolving, so an ORM test does not need URLs to verify classification. - `IndexNowKit::create()` rejects combining a custom `submitter:` with `transport:`, `debounce:`, `throttle:` or `normalizer:`, because a custom submitter builds its own pipeline. Pass those to your submitter instead. - `indexnowkit/testing` ships a mock IndexNow server for end-to-end runs through a real PSR-18 client: `php -S 127.0.0.1:8089 vendor/indexnowkit/testing/resources/mock-server/router.php`, with scenarios selected by an `X-Mock-Scenario` header (`ok200`, `pending202`, `forbidden403`, `ratelimit429`, …), `MOCK_KEYS` for the key files it serves and a request log at `GET /_mock/requests`. The core's own `Psr18TransportTest` runs against a private copy of the same router (`tests/Support/mock-server/`), because the core cannot depend on `testing`. # Writing an adapter For someone who has never read this package's source and wants a working framework adapter by the end of the day. Every section names the core types involved and the conformance scenarios from [docs/spec/03-conformance.md](https://github.com/indexnowkit/spec/blob/main/03-conformance.md) it satisfies. ## 1. Is an adapter the right thing? You do not need a package to use IndexNow. `IndexNowKit::create()` plus a `CallableUrlResolver` covers a single application: ```php $locator = new ArrayResolverLocator(['post' => fn (Post $p) => '/posts/' . $p->slug]); $indexNow = IndexNowKit::create($config, resolver: new AttributeUrlResolver(new AttributeReader(), null, $locator)); ``` An adapter is warranted when other people's applications should get the same behaviour without wiring it. Three shapes exist, and most packages are one of them: - **ORM hook** — the framework has a unit of work and a commit boundary (`indexnowkit/doctrine`). - **CMS hook** — models cannot carry attributes; rules are registered at runtime (WordPress post types, Drupal). - **Framework glue** — container wiring, config, commands, a key-file route (`indexnowkit/symfony-bundle`). ## 2. The 20-minute adapter Everything a minimal adapter needs, in one file, on layer 2 of the kit: `Adapter\ServicesBuilder` describes the graph (your container's pieces as closures, everything else from the core's factories), `Adapter\Services` builds it lazily, `Hook\ObserverHelper` is the never-throwing part of the model hooks. It passes A01, A04, A07 and H01–H03; the core keeps this exact class under test (`tests/Unit/Adapter/TwentyMinuteAdapterTest.php`). ```php final class IndexNowIntegration { public readonly Services $services; private readonly ObserverHelper $hooks; /** @param array $frameworkConfig the raw config array, your own blocks included */ public function __construct(array $frameworkConfig, ?string $environment, LoggerInterface $logger, ?RouteUrlResolverInterface $router = null) { // Never throws: an invalid value is one critical log line and a disabled Config until it is fixed. $config = (new ConfigFactory(ownedOptions: ['myfw.route_prefix'], checkCommand: 'myfw indexnow:check'))->load($frameworkConfig, $environment, $logger); $builder = (new ServicesBuilder($config, $logger)) ->httpClientLocator(fn (string $id): object => $this->service($id) ?? throw new RuntimeException($id)) ->debounceStore(fn (Services $s): DebounceStoreInterface => DebounceStoreFactory::fromConfig($s->config, fn (string $id) => $this->cache($id))) ->resolverLocator(new ArrayResolverLocator([], locate: fn (string $id) => $this->service($id), hint: 'a service id')); if ($router !== null) { $builder->router($router); } $this->services = $builder->build(); // no IO: nothing is built before it is used $this->hooks = new ObserverHelper($this->services->kit(), $logger); } /** Model save hook. */ public function onSaved(object $model, array $changedFields): void { $urls = $this->hooks->guard($model, static fn (ObjectChangeHandler $changes): array => $changes->updated($model, $changedFields)); $this->hooks->deliver($urls ?? []); } /** Model delete hook, before the row disappears: resolve now, deliver once it is gone. */ public function onDeleting(object $model): void { $urls = $this->hooks->guard($model, static fn (ObjectChangeHandler $changes): array => $changes->deleted($model)); $this->hooks->rememberDeletion($model, $urls ?? []); } public function onDeleted(object $model): void { $this->hooks->deliver($this->hooks->takeDeletion($model) ?? []); } /** End of the unit of work: after the response was sent, if the platform allows it. Never throws. */ public function onShutdown(): void { $this->services->flushIfCollected(); } /** GET /{key}.txt */ public function keyFileResponse(string $path, string $host): ?array { $body = $this->services->keyFileResponder()->bodyForPath($path, $host); return $body === null ? null : [$body, $this->services->config->keyFileHeaders()]; } /** How your framework resolves `debounce.store` to a PSR-16 cache and `#[IndexNow(resolver: ...)]` / `http.client` ids to services. */ private function cache(string $id): CacheInterface { /* your container */ } private function service(string $id): ?object { /* your container; null = unknown */ } } ``` Everything below is refinement of those six methods. What the builder did not get comes from the factories of layer 1: `Http\TransportFactory::lazy()` (`http.client`, through your locator), `Debounce\DebounceStoreFactory`, `Dispatch\DispatcherFactory` (`dispatch`; give `queueFactory()` a closure for your queue), `fromConfig()` on `Collector`, `TokenBucket`, `AttributeUrlResolver` and `KeyFileResponder`. Override any node with `transport()`, `submitter()`, `dispatcher()`, `urlResolver()`, … and every dependent node uses the replacement; `build()` throws `ConfigurationException` for what is statically wrong (a `debounce.store` id without a store, a queue mode without a queue). `Services` also gives you `checker()` (add your lines with `checks()`), `submitterFactory()` for the commands, `rules()` for rules registered at runtime, and `hasCollected()`/`flushIfCollected()` for the request-end hook. The parity between the two layers is a test in the core (`ServicesParityTest`). A container that describes services (Symfony, Laravel) stays on layer 1 and calls the same factories service by service: its service ids and bindings are its public API, and a builder would hide them. `IndexNowKit::create()` is the plain-PHP form of the same graph. ### Optional packages `indexnowkit/sitemap` is `suggest`ed, not required: an adapter must work without it and say so where the user looks. The recipe, the same in the three reference adapters: - **One predicate per adapter, `Adapter\OptionalPackage`**: `new OptionalPackage('indexnowkit/sitemap', SitemapReader::class, 'sitemap', $installed)` — `installed()` is `class_exists()` of the marker unless the adapter passes an override (`null` = detect; the bundle's `sitemapInstalled` constructor argument, a Laravel container binding under `IndexNowKitServiceProvider::SITEMAP_PACKAGE`, the Yii2 component's `sitemapInstalled` property). No statics: the override travels with the adapter's own configuration. `notInstalledMessage()`, `checkLine()`, `checkLevel()` and `check()` are the three texts below, written once. - **Separate classes behind it**: every file with a `use IndexNowKit\Sitemap\*` is instantiated only when the predicate holds (`\Sitemap\SitemapServices` that registers the reader, the spool check and the runner; `\Console\SitemapCommand`). A `::class` constant on an absent class is safe; `SitemapConfig::OPTIONS`, `SitemapReader::MAX_*` or `Sitemap\Console\Definitions` in a file that is loaded without the package are a fatal. - **A stub command with the same name** (`SitemapNotInstalledCommand`, or the Yii action) that ignores its arguments, prints `indexnowkit/sitemap is not installed: composer require indexnowkit/sitemap` and exits `ExitCode::FAILURE`: a cron that ran `sitemap` before the package went optional gets a sentence, not "command not found". - **`OptionalPackage::check($block, $defaults)`** in the checker (a `Check\StaticCheck`): `sitemap: not installed (composer require indexnowkit/sitemap)` at level ok when the block is absent or equal to the defaults the adapter ships, or `sitemap: not installed, the sitemap block in the configuration is ignored (composer require indexnowkit/sitemap)` at level warning when the application configured a block nothing reads. That line is the only place the absence is mentioned: no log line at boot or on a request. - **`SitemapConfig::loadOrDisabled($block, $logger, $checkCommand)`** (from `indexnowkit/sitemap`) builds the sitemap configuration at runtime: an invalid block is one `critical` line naming the error and your check command, and a disabled configuration — nothing throws from the container. - **`ConfigFactory(ignoreBlocks: ['sitemap'])`** without the package (and `...SitemapConfig::OPTIONS` in `ownedOptions` with it), so a configuration written for the package does not warn as "unknown option" once the package is gone. `ownedOptions` stays dotted: a bare `sitemap` in it would hide every typo inside the block. ## 3. The component graph ``` Transport -> Client -> Submitter -> Collector + Dispatcher -> IndexNowKit ^ ^ KeyProvider UrlResolver <- AttributeReader ``` `IndexNowKit::create()` builds all of it with sensible defaults; every argument is optional and named, and parameter **names** are part of the compatibility promise, so always pass them by name. In a container, build the same graph service by service — that is exactly what the Symfony bundle does, and nothing in the library requires the facade. Two rules when substituting pieces. A custom `submitter:` brings its own pipeline, so combining it with `transport:`, `debounce:`, `throttle:` or `normalizer:` is rejected instead of silently ignored. And wrap the transport in `Http\LazyTransport` so a request that submits nothing never pays for client discovery and never fails on a missing PSR-18 client: ```php $transport = new LazyTransport(fn () => Psr18Transport::discover(timeout: $config->httpTimeout)); ``` ## 4. Configuration Map your framework's config file onto `Config::fromArray()`. `Config::OPTIONS` is the canonical list of keys the core owns; `Config::unknownOptions($data, $allowed)` reports typos in the rest, so `debounce.per_urls` does not pass silently. Strip your own blocks before handing the array over. `dispatch` is a free identifier the core validates and reports but never acts on — the adapter decides what `sync`, `queue` or `messenger` mean. Feed `environment` from your framework's environment name to get the non-production dry-run safety net. Full details in [configuration.md](configuration.md). If your config can only be validated at runtime (environment placeholders), do not let a bad value throw from a save hook. `Adapter\ConfigFactory` is that path, declared once per adapter: ```php $factory = new ConfigFactory( ownedOptions: ['queue.connection', 'queue.delay', 'key_file.path', ...SitemapConfig::OPTIONS], // dotted keys only dispatchModes: ['queue', 'sync', 'none'], // [0] is what `dispatch: auto` may not be: see autoDispatch autoDispatch: static fn (): string => $queueExists ? 'queue' : 'sync', needBaseUrl: ['queue'], // a worker has no request to take the host from defaults: ['dispatch' => 'auto', 'debounce' => ['store' => 'cache']], // scalars and blocks of scalars, never lists validate: static fn (Config $c): ?string => $c->dispatch === 'queue' && !$queueExists ? 'the queue component is not configured' : null, checkCommand: 'myfw indexnow:check', ); $config = $factory->load($raw, $environment, $logger); // runtime: warning on unknown keys, critical + disabled on an error $config = $factory->build($raw, $environment); // check command, tests: throws ConfigurationException ``` The merge is deliberate: a top-level raw key replaces the default, the known blocks (`http`, `debounce`, `key_file`, `throttle`, `retry`, `batch`, `logging`) and your owned blocks merge key by key, lists (`engines`, `hosts`) come from the raw array untouched. `key_file.enabled`, `key_file.cache_max_age`, `debounce.store` and `http.client` are core options: read them from the `Config`, do not carve them out. ## 5. How your framework says "this object has a public page" | Model | Core piece | |---|---| | PHP attributes on the class | `Attribute\AttributeReader` (the default) | | rules registered in code, per class or per object | `Attribute\RuleRegistry` | | your own metadata source | implement `Attribute\AttributeReaderInterface` | | the object knows its own URL | `#[IndexNowUrl]` on the method, or `Url\CallableUrlResolver` | | attributes behind `__get()` / an array (Eloquent, CMS records) | implement `Attribute\SubjectReaderInterface`, register it once with `ParamExtractor::registerReader()` | `RuleRegistry` decorates any reader, so a CMS adapter keeps attribute support for free: ```php $registry = new RuleRegistry(); // wraps AttributeReader by default $registry->register(Post::class, [new IndexNow(route: 'posts.show', params: ['post' => 'self'])], new IndexNowDefaults(when: 'isPublished')); $registry->registerFor(CmsPage::class, fn (CmsPage $p): ?RuleSet => $rulesFor($p)); // null = fall through ``` Whatever the source, every path should end at `GuardedUrlResolver`, which is the only never-throwing entry point. ## 6. URLs Implement `Url\RouteUrlResolverInterface` for your router. It is deliberately two methods, so the core can re-extract parameters per locale and pin a host per rule: ```php public function locales(array|string $locales): array; // list; [null] = no locale dimension public function generate(string $route, array $params, ?string $locale = null, ?string $host = null): string; ``` - `locales('current')` returns `[null]`; `'all'` returns every locale your framework has enabled; an explicit list is returned as given. An empty list means "no locale dimension" and must become `[null]`. - `generate()` returns an **absolute** URL. Outside a request there is no host to inherit, so fall back to `$config->baseUrl`; when `$host` is given, prefer `$config->baseUrlFor($host)` and fall back to `https://$host`. - Wrap your router's exceptions in `ConfigurationException` with the route name in the message. A missing parameter is the most common attribute mistake and the message is what the user will see. - Parameters arrive already extracted and coerced. An object value means route model binding (`params: ['post' => 'self']`); decide in the bridge how your router consumes it. Without a router, use `Url\ArrayResolverLocator` to serve `#[IndexNow(resolver: ...)]`, or let models expose `url:`/`urls:` rules. Replace `Url\UrlNormalizerInterface` only to change canonical form — stripping tracking parameters, enforcing a trailing-slash policy, mapping hosts. Implementations must throw `InvalidUrlException` and nothing else, or they break the never-throw contract of `submit()`. ## 7. Hooking model changes `Url\ObjectChangeHandler` is the piece to build on. It combines rule lookup, per-rule event classification and guarded resolution, and never throws: an invalid rule set or a failing resolver is logged and yields nothing. ```php $changes = $indexNow->changes(); // or new ObjectChangeHandler($reader, $guarded, $logger) $guarded = $indexNow->resolver(); // the GuardedUrlResolver behind it, for explain() and resolveRule() $changes->created($model); // list $changes->updated($model, $changedFields, $changeSet); $changes->deleted($model); ``` Two levels exist because ORMs differ. Hooks that run **before** the write, where ids do not exist yet, collect `RuleEvent`s first and resolve them later: ```php $events = $changes->updatedEvents($model, array_keys($changeSet), $changeSet); // list // ... the write happens ... foreach ($events as $ruleEvent) { $urls = $changes->resolve($model, $ruleEvent); } ``` Hooks that run **after** the write (observers, save hooks) call `created()` / `updated()` / `deleted()` directly. Three things decide correctness: - **Deletions must be resolved while the object still has its identifiers and old state.** That includes a rule whose `when` just turned false: `updatedEvents()` returns it as `Event::Deleted`, and it must be resolved before the write, not after. - **Supply both `$changedFields` and `$changeSet` when you have them.** The change set (`field => [old, new]`) is what makes the old-state visibility exact instead of heuristic. See [attribute-reference.md](attribute-reference.md#reconstructing-w_before). - **Never let a hook throw into the host application.** A typo in an attribute must not break a checkout. Satisfies A03, A04, A07, A08, A09, A10, A12. ### Example: an Eloquent-style observer ```php final class IndexNowObserver { public function __construct(private readonly IndexNowKit $indexNow) {} public function created(Model $model): void { $this->collect($this->indexNow->changes()->created($model)); } public function updated(Model $model): void { $changeSet = []; foreach ($model->getChanges() as $field => $new) { $changeSet[$field] = [$model->getOriginal($field), $new]; } $this->collect($this->indexNow->changes()->updated($model, array_keys($changeSet), $changeSet)); } public function deleting(Model $model): void // before the row disappears { $this->collect($this->indexNow->changes()->deleted($model)); } private function collect(array $resolved): void { $this->indexNow->collect(ResolvedUrl::urls($resolved)); } } ``` Register it so its URLs are handed over only **after** the surrounding transaction commits: resolve synchronously (the old state is live), hand off through the framework's after-commit hook (`Connection::afterCommit()` in Laravel). If your framework has no such hook, use the next section. ## 8. Commit safety URLs must not leave before the outermost transaction commits, or a rolled-back write is announced to search engines. `Transaction\TransactionStaging` lives in the core precisely so every adapter solves this the same way. ```php $staging = new TransactionStaging(sink: fn (array $urls) => $indexNow->collect($urls), logger: $logger); $staging->stage($scope, $urls); // inside an open transaction $staging->commit($scope); // real COMMIT: hands the URLs to the sink $staging->discard($scope); // ROLLBACK, or a commit that threw: drops them, logged at debug ``` `$scope` is any object whose identity outlives the transaction — the native database connection is the usual choice. Entries are held in a `WeakMap`, so a forgotten scope does not leak. `hasPending()` and `pendingCount()` are there for diagnostics. Where the real commit signal comes from differs per framework: a DBAL driver middleware (Doctrine), `Connection::afterCommit()` (Laravel — its transaction manager already drops callbacks of a rolled-back savepoint, so the Laravel adapter needs no staging of its own; `ShouldHandleEventsAfterCommit` is *not* used, because a deferred `updated` handler runs after `syncOriginal()` and loses the old values), `transaction.on_commit` (Django). When the framework offers none, use `Transaction\VerifyingStaging` instead of guessing. Satisfies A01, A02, A05. ### 8a. No commit signal at all: verify on commit Yii2 fires commit/rollback events only for the outermost transaction and nothing for savepoints; Yii3's `yiisoft/db` fires nothing. `Transaction\VerifyingStaging` holds URLs together with a *verifier*, a closure that re-reads the row by primary key and says whether the change actually landed (created/updated: the row exists with the new values, `VerifyingStaging::rowMatches($row, $expected)`; deleted: no row): ```php $staging = new VerifyingStaging($logger); // in the ORM event, when a transaction is open: $staging->stage($connection, fn (): bool => $this->rowMatches($record, $written), $urls, Post::class . '#' . $id); // when the data layer says the transaction ended (commit event), or at the end of the request when it says nothing: $indexNow->collect($staging->flush($connection)); // runs the verifiers, drops what did not land (logged at debug) $staging->discard($connection); // on a rollback event: nothing to verify ``` One primary-key lookup per staged subject, only for changes inside an explicit transaction (autocommitted changes go straight to the collector). A change that did not land drops every URL it produced, including `via` pages and the old URL of a renamed page: announcing "deleted" for a page that still exists is the one outcome to avoid. A verifier that throws counts as landed (a stale URL costs one crawl, a lost one costs the update) and is logged at warning. Satisfies A02, A05, A05b, A05c without touching the connection configuration; the Yii adapters are the reference. ## 9. The unit of work `Collector\CollectorInterface` buffers normalized URLs for one HTTP request, console command or queue message, and `IndexNowKit::flush()` drains it into the dispatcher exactly once. Call `flush()`: - after the response has been sent, where the platform allows it (`kernel.terminate`, `fastcgi_finish_request`); - at the end of a console command; - after each handled queue message. In long-running runtimes call `CollectorInterface::reset()` between requests. The default `Collector` logs a `warning` when a reset discards a non-empty buffer, which is the signal that a unit of work ended without a flush. Do not swallow it. Replace the interface for a durable outbox or a per-tenant buffer. Satisfies A06, H06. ## 10. Delivery `Dispatch\DispatcherInterface` has one method and must never throw into user code. `SyncDispatcher` sends inline, `CallableDispatcher` hands the list to any queue, `NullDispatcher` drops it (`dispatch: none` — collect, never send). The worker recipe is the same everywhere: `submit()`, then `Result::retryableUrls($results)`, then `RetryPolicy::delayAfter($results, $attempt)`, then re-enqueue. Which statuses are final and which are retryable is in [retries-and-queues.md](retries-and-queues.md). Satisfies A14, C13. A worker has no request context, so `base_url` must be set or every relative URL is dropped. ## 11. Keys and the key file `Key\KeyProviderInterface` has four methods and is called on the submission path, so implementations must be cheap and must never throw for an unknown host: ```php public function keyFor(string $host): ?string; // null = unmanaged, URLs are skipped public function keyLocationFor(string $host): ?string; public function isKnownKey(string $key, ?string $host = null): bool; // serve /{key}.txt? public function managedHosts(): array; // diagnostics; empty when unknown ``` `StaticKeyProvider::fromConfig($config)` covers config-backed setups including `strict_hosts`. For a database-backed multi-tenant install, implement the interface and cache per request. **Honour the `$host` argument of `isKnownKey()`**: without it, tenant A's key file is served on tenant B's host, which lets one tenant claim ownership signals on another's domain. `null` means "any managed host" and is only for single-site adapters and CLI diagnostics. Serving the file is `Key\KeyFileResponder`, so no adapter reimplements the matching: ```php $responder = new KeyFileResponder($keys, $config->serveKeyFile); $body = $responder->bodyForPath($request->getPath(), $request->getHost()); // or bodyForKey() if your router if ($body === null) { return $this->notFound(); } // extracted {key} already return $this->response($body, 200, KeyFileResponder::headers($maxAge)); ``` `KeyFileResponder::PATH_PATTERN` is the request-path regex (group 1 is the key) for routers that match by pattern, `CONTENT_TYPE` is `text/plain; charset=utf-8`, and `DEFAULT_MAX_AGE` is 300 seconds — short on purpose, because a cached old key file turns every submission into a 403 after a rotation. Serve 200 with no redirect, 404 otherwise. `Key\KeyGenerator::generate($length, $hex)` produces CSPRNG keys, 32 hex characters by default; pass `hex: false` for the full `[A-Za-z0-9]` alphabet. A `key:generate --write-env` style command is the first thing users run. Satisfies H01–H03. ## 12. Transport `Http\TransportInterface` is two methods over any HTTP stack — `wp_remote_post()`, a framework client, raw curl: ```php public function post(string $url, string $json, array $headers = []): Response; public function get(string $url): Response; ``` Rules: never throw on an HTTP status code, throw `Http\Exception\TransportException` for network failures and timeouts, cap the body you read (`Psr18Transport` uses 2 KiB for POST diagnostics and 50 MiB for GET, a generous cap for the largest documents consumers of the transport read). Parse `Retry-After` with `Response::parseRetryAfter($header)` so every adapter interprets delta-seconds and HTTP-dates identically and applies the same clamp. Configure no redirects and a timeout. Implement `Http\StreamingTransportInterface` too when your stack can read a response body in chunks (`download(string $url, $sink): Response` writes the body to a stream resource and returns an empty-bodied `Response`). Consumers that read large documents (the add-on packages) then never hold a document in memory; with a plain `TransportInterface` they buffer each document once through `get()`. `LazyTransport` and `Testing\FakeTransport` implement both. ## 13. Debounce, throttle, clock `MemoryDebounceStore` is per process and bounded; `Psr16DebounceStore` shares the window across processes through any PSR-16 cache; `NullDebounceStore` disables it. Wire your framework's cache to the PSR-16 one by default for web applications, and memory for CLI and tests. A debounce store may throw: the submitter treats a failing read as "nothing is recent" and a failing write as "window not recorded", logs a warning, and delivers anyway. Preserve that fail-open behaviour in your own store. `TokenBucket` blocks with `usleep()` per process. In a web request `NullThrottle` is often the better default, with the real rate limiting in the queue worker. Both take a `Psr\Clock\ClockInterface`, so tests use `FrozenClock`. ## 14. Diagnostics users will ask for Ship six commands. They are what turns "it does not work" into a self-service answer, and their bodies are the `indexnowkit/console` package (`Console\*Runner`, rendering to a `Symfony\Component\Console\Style\SymfonyStyle`; Laravel's `OutputStyle` is one; require it — the core itself does not depend on `symfony/console`). A framework command parses its arguments and calls the runner; every framework prints the same thing. | Command | Runner | What the adapter supplies | |---|---|---| | `check` | `Console\CheckRunner` | a closure that builds `Config` from the raw configuration (throws `ConfigurationException`); `Check\CheckInterface` services for adapter wiring (is the ORM hook active? is the queue routed?) and the checks of the add-on packages | | `submit ...` | `Console\SubmitRunner` | — | | `submit- [ids]` | `Console\SubmitSubjectsRunner` + `SubmitSubjectsOptions` | a `Console\SubjectLoaderInterface`: class resolution (FQCN or the framework's short name), objects by id, first N objects; `byIds()` / `all()` receive the `Event` so `deleted` can include soft-deleted rows | | `explain ` | `Console\ExplainRunner` | the same loader | | `key:generate` | `Console\KeyGenerateRunner` | the default env file path | The sixth command, which submits the site's own URL list, is an add-on package (see the family table in the README): its runner, options and check live there, and its `docs/adapters.md` says how to wire the command. Shared by all of them: `Adapter\SubmitterFactoryInterface` (`SubmitterFactory`: the separate submitter `--force` and `--dry-run` build, `SubmitterFactory::choose()` picks it or the application's), `Console\ResultFormatterInterface` (`ResultRenderer`: table or `--json`; an application replaces it to match its own CLI), `Submission\ResultSummary` (a run that submits in many batches folds results into counts) and `Console\Vocabulary` (the words that differ: "entity" / "model", `bin/console` / `php artisan`, where the configuration lives). Expose the three interfaces under stable service ids so an application can decorate them, and expose the runners too: a tenant loop over `SubmitSubjectsRunner` is a ten-line application command. `Checker` never throws and covers configuration, key files and a live probe; `CheckRunner` prints its report. The adapter-specific lines are `CheckInterface` services tagged for the checker, not special cases in the command. Result listeners (`SubmitterInterface::addListener()`) feed an admin log or a profiler panel. Register the listener on the same submitter instance the application uses, and forward `addListener()` in any decorator. ## 15. The error contract | Situation | Behaviour | |---|---| | invalid `Config` | throws `ConfigurationException` at construction | | invalid rule declaration read through `AttributeReaderInterface` | throws `ConfigurationException` | | invalid rule declaration read through `ObjectChangeHandler` / `GuardedUrlResolver` | logged at `error`, yields no URLs | | resolver failure (missing accessor, router error) in a hook | logged at `error`, yields no URLs | | URL that cannot be submitted | `InvalidUrlException` inside the normalizer, caught by `Submitter`, `warning` + `skipped` result | | HTTP status of any kind | never throws; a `Result` with a `Reason` | | network failure | `TransportException` inside the transport, converted to a retryable `failed` result | | debounce store, throttle, listener or dispatcher failure | logged, delivery continues | | programming errors (empty batch, bad key length) | `InvalidArgumentException` | The golden rule: **nothing reaching a lifecycle hook may throw into the host application.** ## 16. Testing your adapter Use `IndexNowKit\Testing` (`FakeTransport`, `ArrayLogger`, `FrozenClock`, `RecordingDispatcher`) — see [testing.md](testing.md). Assert classification through `ObjectChangeHandler::*Events()` before any URL exists, and delivery through `RecordingDispatcher`. For the HTTP and command scenarios, parse your framework's response or output and hand it to `Testing\Conformance\KeyFileAssertions` (H01–H03: status, content type, `Cache-Control` by directive, `Vary: Host` only with a hosts map) and `Testing\Conformance\CheckOutputAssertions` (H04–H05: exit code with the output as the failure message, the ready line, the key file hint), so your tests do not carry a copy of the core's phrases. Both, the conformance kits and the mock server are the `indexnowkit/testing` package (`require-dev`); the core ships only the four PHPUnit-free doubles. Then work through the conformance scenarios with the kits of `indexnowkit/testing` (`Testing\Conformance\CoreConformanceTestCase`, `OrmConformanceTestCase`, see [testing.md](testing.md)): C01–C22 for anything that talks to the protocol, A01–A21 for an ORM adapter, H01–H06 for a framework adapter. Declare in your README which scenarios do not apply to your framework and why — A13 (bulk operations bypass hooks) is a documented limitation everywhere, not a failure. ## 17. Packaging Name it `indexnowkit/`, require `indexnowkit/core ^0.5`, keep the framework itself in `require` and the optional pieces in `suggest` (`indexnowkit/sitemap ^0.1.1` for the `sitemap` command and its `Definitions`, wired as in §2 "Optional packages"; keep it in `require-dev` so the tests cover both states). Run a version matrix in CI over the framework's supported majors and LTS releases, static analysis at the maximum level, and publish EN plus RU READMEs following the family table used here. The Definition of Done is in [docs/spec/91-roadmap.md](https://github.com/indexnowkit/spec/blob/main/91-roadmap.md). ## 18. Reference adapters The bundle and the Laravel package sit on layer 1 (the static factories and `Adapter\ConfigFactory`, one service or binding per node, because those ids are their public API); the Yii2 component sits on layer 2 (`Adapter\ServicesBuilder`, the graph described once, the pieces exposed as delegates). All of them share `Hook\ObserverHelper` in the observers, `Retry\WorkerOutcome` in the queue jobs and `Console\Definitions` (`indexnowkit/console`) in the commands. | Section | `doctrine` | `symfony-bundle` | `laravel` | `yii2` | |---|---|---|---|---| | layer | — | 1 (services) | 1 (bindings) | 2 (`ServicesBuilder`) | | component graph | `src/IndexNowDoctrine.php` | `src/DependencyInjection/IndexNowKitLoader.php` | `src/IndexNowKitServiceProvider.php` | `src/IndexNowComponent.php` (`services()`) | | configuration | — | `src/DependencyInjection/{IndexNowKitConfiguration,ConfigFactory}.php` | `config/indexnow.php`, `src/Config/ConfigFactory.php` | `src/Config/ConfigFactory.php` | | router bridge | — | `src/Url/SymfonyRouteUrlResolver.php` | `src/Url/LaravelRouteUrlResolver.php` | `src/Url/YiiRouteUrlResolver.php` | | resolver lookup | — | `src/Url/ResolverLocatorFactory.php` (core `ArrayResolverLocator`) | in the provider (core `ArrayResolverLocator`) | in the component (core `ArrayResolverLocator`) | | model change hooks | `src/IndexNowListener.php` | via the Doctrine package | `src/Eloquent/IndexNowObserver.php` (`ObserverHelper` + `afterCommit()`) | `src/ActiveRecord/IndexNowObserver.php` (`ObserverHelper` + staging), `IndexNowBehavior.php` | | commit safety | `src/Middleware/*` | `src/Doctrine/StagingSink.php` | Laravel's `afterCommit()` | core `VerifyingStaging` | | unit of work | — | `src/EventListener/FlushListener.php` | `terminating()`, `JobProcessed` | `EVENT_AFTER_SEND`, `EVENT_AFTER_REQUEST` | | delivery | — | `src/Messenger/*` (`WorkerOutcome`) | `src/Queue/*` (`WorkerOutcome`) | `src/Queue/*` (yii2-queue, `WorkerOutcome`) | | key file | — | `src/Controller/KeyFileController.php`, `config/routes.php` | `src/Http/KeyFileController.php` | `src/Http/KeyFileController.php` | | diagnostics | — | `src/Command/*` (`Definitions`), `src/DataCollector/*` | `src/Console/*` (`Definitions`), `src/Check/*` | `src/Console/IndexNowController.php` (`Definitions`), `src/Check/*` | | subject reader | — | — | `src/Eloquent/EloquentSubjectReader.php` | `src/ActiveRecord/ActiveRecordSubjectReader.php` | ## 19. Compatibility What the core guarantees, what is excluded, and how to ask for a new extension point instead of reaching into `@internal`: [bc.md](bc.md). ## 20. Definition of Done for an adapter - [ ] `Adapter\ConfigFactory` declared with dotted `ownedOptions` (plus `SitemapConfig::OPTIONS` when the package is installed, `ignoreBlocks: ['sitemap']` when it is not); a regression test that a typo inside an owned block (`key_file.enabld`) is warned about; an invalid runtime value disables IndexNow with one `critical` line and never throws from a hook. - [ ] The graph is built through the factories (`Http\TransportFactory`, `Debounce\DebounceStoreFactory`, `Dispatch\DispatcherFactory`, `fromConfig()`); no copied `match` over `debounce.store`, no own "not a PSR-18 client" text, no own class-name resolution (`Console\ClassNameResolver`). - [ ] `#[IndexNow(resolver: ...)]` through `ArrayResolverLocator(locate:, hint:)`; the resolver is `GuardedUrlResolver`. - [ ] Hooks over `Hook\ObserverHelper` (guard, deliver, remembered deletions; no own `WeakMap`, no own "cannot resolve" text); deletions resolved before the row disappears; a commit boundary (`afterCommit`, `TransactionStaging`, `VerifyingStaging`). - [ ] A queue job over `Retry\WorkerOutcome` (retryable vs final, the three log lines) plus your framework's action; or a runtime-assembled container over `Adapter\ServicesBuilder` with `queueFactory()`. - [ ] Flush at the end of every unit of work (request, command, queue message); `Collector::reset()` in long-running runtimes. - [ ] `KeyFileResponder::fromConfig()` + `Config::keyFileHeaders()` on a route without session or CSRF; H01–H03 green. - [ ] Six commands over the runners of `indexnowkit/console` (`sitemap` from `indexnowkit/sitemap`), their inputs from `Console\Definitions` / `Sitemap\Console\Definitions` (no own option descriptions), `check` with your `CheckInterface` lines plus `Check\DebounceStoreCheck` (with a probe) and `Sitemap\Check\SitemapSpoolCheck`. - [ ] `indexnowkit/sitemap` in `suggest` and `require-dev`, behind one predicate (§2 "Optional packages"): without it the `sitemap` command is a stub that explains what to install and exits 1, `check` prints the `StaticCheck` line, a `sitemap` block in the configuration warns about nothing, every other command works, and nothing is logged at boot; a test set with the predicate forced to false. - [ ] `indexnowkit/testing` in `require-dev`; conformance kits green (C01–C22, A01–A21 for an ORM, H01–H06 through `Testing\Conformance\KeyFileAssertions` and `CheckOutputAssertions`); undocumented scenarios named in the README. - [ ] CI matrix over the framework's supported majors, phpstan level 9 on every flavour, EN + RU README with the family table, `docs/troubleshooting.md`, a changelog with migration notes. # Backward compatibility `indexnowkit/core` follows SemVer. **Before 1.0, minor versions may contain breaking changes**; every one is listed under "Changed" in [CHANGELOG.md](https://github.com/indexnowkit/php/blob/main/packages/core/CHANGELOG.md) with the migration. After 1.0 the rules below become the promise. This page exists because "public API" is ambiguous for a library whose main audience is other library authors. ## Three tiers | Tier | What it means | Examples | |---|---|---| | **Call** | You call it. Signatures do not change incompatibly; new parameters are only appended with defaults. | `IndexNowKit`, `Config` (including the static `serveKeyFileFrom()`), `Submitter`, `Client`, `Result`, `Checker`, `KeyGenerator`, `KeyFileResponder`, `RetryPolicy`, `ObjectChangeHandler`, `GuardedUrlResolver`, `RuleRegistry`, `Transaction\VerifyingStaging`, `Adapter\SubmitterFactory`, `Submission\ResultSummary`, `Adapter\ConfigFactory`, `Adapter\ServicesBuilder`, `Adapter\Services`, the factories (`Http\TransportFactory`, `Debounce\DebounceStoreFactory`, `Dispatch\DispatcherFactory`, every `fromConfig()`), `Check\DebounceStoreCheck`, `Check\StaticCheck`, the writers of `Check\CheckReport`, `Hook\ObserverHelper`, `Retry\WorkerOutcome`, `Submission\NullSubmissionStore`, the four test doubles of `Testing\` | | **Implement** | You implement it, and the core calls you. Methods are not added without a major version. | `TransportInterface`, `StreamingTransportInterface`, `Url\RuleAwareUrlResolverInterface` (until 1.0 a method may still be appended in a minor), `Check\CheckInterface`, `KeyProviderInterface`, `UrlNormalizerInterface`, `UrlResolverInterface`, `DebounceStoreInterface`, `ThrottleInterface`, `DispatcherInterface`, `Attribute\SubjectReaderInterface`, `Adapter\SubmitterFactoryInterface`, `Submission\SubmissionStoreInterface` (new in 0.8, see [submission-store.md](submission-store.md)), `Attribute\Param\Condition` and `FieldCondition` (new in 0.8, the `when` guards); the three new interfaces live through one minor unchanged before 1.0 | | **May grow** | Interfaces the core also implements for you, where a new method may appear in a minor. Extend the shipped class rather than implementing the interface from scratch. | `ClientInterface`, `Check\CheckerInterface`, `SubmitterInterface`, `CollectorInterface`, `AttributeReaderInterface`, `RouteUrlResolverInterface`, `ResolverLocatorInterface` | The "may grow" tier is the honest label for interfaces that are still learning what adapters need. If you implement one directly, pin `^0.8.0` rather than `^0.8` and read the changelog before upgrading. Decorating a shipped implementation (`RetryingSubmitter` decorates `Submitter`, `RuleRegistry` decorates `AttributeReader`) is safe in both directions. `RouteUrlResolverInterface` and `ResolverLocatorInterface` have no shipped implementation to decorate (one per framework adapter): pin `^0.8.0` and read the changelog. ## Named arguments `IndexNowKit::create()` takes thirteen optional arguments after `$config` and will take more. **Parameter names are part of the promise; the order is not.** New parameters are appended, never inserted, and every call should use named arguments: ```php IndexNowKit::create($config, transport: $transport, logger: $logger, resolver: $resolver); ``` The same holds for the constructors of `Config`, `Client`, `Submitter`, `AttributeUrlResolver`, `GuardedUrlResolver`, `TransactionStaging`, `VerifyingStaging`, `RetryPolicy`, `TokenBucket`, `Collector` and `Psr18Transport`: pass anything past the first argument by name. `RuleCompiler` (`compile()`, `fromAttributes()`) and `ParamExtractor` (`extract()`, `read()`, `condition()`, `registerReader()`, `unregisterReader()`) are public static helpers in the same "call" tier: adapters call them to compile their own declarations and to plug in a `SubjectReaderInterface`; their signatures only grow by appended optional parameters. The shipped default implementations are in the "call" tier as well: construct them with named arguments and their public methods stay. That is `Http\LazyTransport` (the default `IndexNowKit::$transport`), `Http\Psr18Transport`, `Key\StaticKeyProvider`, `Url\UrlNormalizer`, `Url\ArrayResolverLocator`, `Url\CallableUrlResolver`, `Url\NullUrlResolver`, `Attribute\AttributeReader`, `Attribute\ChangeClassifier`, `Collector\Collector`, `Debounce\{MemoryDebounceStore, Psr16DebounceStore, NullDebounceStore}`, `Throttle\NullThrottle`, `Dispatch\{SyncDispatcher, CallableDispatcher, NullDispatcher}` and `Clock\SystemClock`. `Config::with()` takes constructor parameter names as keys and rejects unknown ones with a message listing what it accepts. Renaming a `Config` property is therefore a breaking change and appears in the changelog. ## Value objects and enums `Result`, `ResolvedUrl`, `UrlRule`, `RuleSet`, `RuleEvent`, `Http\Response`, `Check\CheckItem`, `Retry\WorkerOutcome`, `Submission\SubmissionRecord` and the attribute classes are `final readonly`. Their properties are read-only public API: reading them is safe, constructing them is safe, and new properties are only appended with defaults. Prefer the named constructors (`Result::ok()`, `Result::skipped()`, `Result::failed()`) over the constructor, so an appended parameter never reaches your call sites. Enums are a special case: **adding a case is not a breaking change** in this library, because the wire protocol and the failure taxonomy grow. | Enum | Adding cases? | |---|---| | `Reason` | yes — always handle unknown cases with a `default` arm | | `Engine` | yes — new IndexNow participants get added | | `Attribute\RuleSource` | yes | | `Event`, `ResultStatus`, `Check\CheckLevel` | no; these are closed sets | A `match` over `Reason` or `Engine` without a `default` will fatal on a new case. Write the default arm. The `Reason` cases and what they mean for a `Result` (`isSkip()`: nothing was sent; `isRetryable()`: a later attempt may succeed by itself): | Case | `isSkip()` | `isRetryable()` | Produced by | |---|---|---|---| | `disabled`, `dry_run`, `debounced`, `no_key`, `invalid_url` | yes | no | the core pipeline | | `noindex`, `robots_disallowed`, `non_canonical`, `redirected` | yes | no | the `verify` package's pre-flight (cases reserved in core 0.8) | | `origin_error` | yes | yes | `verify`: the page could not be fetched | | `invalid_request` (400), `invalid_key` (403), `unprocessable` (422), `unexpected` | no | no | the engine's answer | | `rate_limited` (429), `server_error` (5xx), `transport` | no | yes | the engine's answer or the network | ## Constants These are the values to reference instead of hard-coding, and they are covered by the promise: `Config::MAX_BATCH_URLS`, `Config::DEFAULT_BATCH_MAX_URLS`, `Config::DEFAULT_DEBOUNCE_PER_URL`, `Config::DEFAULT_THROTTLE_PER_MINUTE`, `Config::DEFAULT_HTTP_TIMEOUT`, `Config::PRODUCTION_ENVIRONMENTS`, `Config::OPTIONS`, `Result::NO_ENGINE`, `Client::FORBIDDEN_ESCALATION`, `KeyValidator::MIN_LENGTH`, `KeyValidator::MAX_LENGTH`, `KeyValidator::ALPHABET`, `KeyValidator::PATTERN`, `KeyFileResponder::PATH_PATTERN`, `KeyFileResponder::CONTENT_TYPE`, `KeyFileResponder::DEFAULT_MAX_AGE`, `Http\Response::MAX_RETRY_AFTER`, `Psr18Transport::POST_BODY_LIMIT`, `Psr18Transport::GET_BODY_LIMIT`, `UrlNormalizer::MAX_URL_LENGTH`, `UrlNormalizer::MAX_HOST_LENGTH`, `UrlNormalizer::MAX_LABEL_LENGTH`, `ParamExtractor::SELF`, `Version::VERSION`. Enums (`ResultStatus`, `Reason`, `Event`, `Engine`, `Check\CheckLevel`, `Attribute\RuleSource`, `Attribute\Param\Placeholder`) and the value objects of the rule model (`Attribute\UrlRule`, `RuleSet`, `RuleEvent`, `Attribute\Param\{Accessor, Value, Formatted, Call}` and the condition `Attribute\Param\Equals`, `Url\ResolvedUrl`) are public API: their public properties are read by adapters and their constructors only grow by appended optional parameters. Their **values** may change in a minor when the protocol or a safety limit changes; the constants themselves will not disappear. ## Exceptions Every exception implements `Exception\IndexNowException`, so `catch (IndexNowException $e)` is the stable form. `ConfigurationException`, `InvalidUrlException`, `InvalidArgumentException` and `Http\Exception\TransportException` keep their meanings. `ConfigurationException` and `InvalidUrlException` extend `Exception\InvalidArgumentException`, which extends PHP's `\InvalidArgumentException`, so both `catch (Exception\InvalidArgumentException)` and `catch (\InvalidArgumentException)` see them. Exception **messages** are not API: they are written for humans and get improved. Match on the class, or on `Result::$reason`, never on message text. ## What is not covered - Anything marked `@internal` in a docblock. Today that is `Url\Punycode`, `Transaction\StagingFrame`, `Attribute\IndexNow::normalizeEvents()`, `Collector::reportLeak()` and the constructor of `Adapter\Services` (built by `ServicesBuilder::build()`). - Private and protected members of `final` classes, which is all of them: the library has no inheritance points by design, only interfaces. - Log message texts. They are documented in [operations.md](operations.md) so you can grep them, and they are improved between versions. Alert on `Reason` values and log **levels**, not on wording. - Anything under `tests/`, including fixtures and the mock server copy. The published test doubles live in `IndexNowKit\Testing` and **are** covered. The conformance kits (`Testing\Conformance\CoreConformanceTestCase`, `OrmConformanceTestCase`) and the assertion helpers are the `indexnowkit/testing` package since 0.7.0, with their own [bc.md](../testing/bc.md): driver methods only grow by appended methods with a default implementation, a scenario is only added, never removed, in a minor. - The exact set of `Result` objects a single `submit()` call returns. Grouping by host and batching are implementation details of throughput; use `Result::allUrls()`, `Result::retryableUrls()` and `Result::urlsWhere()` instead of indexing into the list. ## Deprecations A deprecated member keeps working for at least one minor version, carries a `@deprecated` tag naming the replacement, and is listed in the changelog. Currently deprecated: | Since | Member | Use instead | |---|---|---| | 0.4.0 | `serve_key_file` (`Config::fromArray()`, `fromEnv()`: `INDEXNOW_SERVE_KEY_FILE`) | `key_file.enabled` / `INDEXNOW_KEY_FILE_ENABLED`; the explicit `serve_key_file` still wins while both exist | Removed after their deprecation window: `Result::urlsOf()` (deprecated 0.2.0, removed 0.4.0). Moved out of the core without a deprecation window (the pre-1.0 rule): in 0.4.0 `IndexNowKit::sitemap()` and everything under `Sitemap\`, now the `indexnowkit/sitemap` package; in 0.7.0 `Testing\Conformance\*` and the assertion helpers (`Testing\KeyFileAssertions`, `CheckOutputAssertions`, `ReadmeAssertions`, now `Testing\Conformance\*` in `indexnowkit/testing`) and everything under `Console\` except `SubmitterFactory*` (now `Adapter\`) and `ResultSummary` (now `Submission\`), now the `indexnowkit/console` package with the FQCN unchanged and its own [bc.md](../console/bc.md). ## Before 1.0 Minor versions may break. The changes made in 0.2.0, 0.4.0, 0.7.0 and 0.8.0 are listed in the changelog (0.5.0 and 0.6.0 were additive); the shape of the breakage to expect is the same: renamed classes as the namespace layout settles, and signatures on the "may grow" interfaces as more adapters land. Application code that only uses the facade, `Config`, the attributes and `Result` has been stable since 0.1 and is expected to stay so. If you need an extension point that does not exist, open an issue rather than reaching into `@internal` or copying a final class. Adapter-driven interface changes are exactly what the pre-1.0 window is for. # Codes of `check` Every line the check command prints carries a stable code (`Check\CheckItem::$code`). The code is what `check --json` consumers, deploy pipelines and alert rules match on; the **text is not API** and gets improved between versions, the same way `Reason` is the identifier of a `Result` and `Result::$error` the sentence. A code names the check, not the outcome: `key_file.status` is `ok` when the key file answers 200 with the right body and `error` when it does not, so a rule written as "fail the deploy when `key_file.status` is not ok" survives a rewording. Codes are added in minor versions when a check is added (a new line in the table below), and never renamed or removed before 1.0 without an entry under "Changed" in the changelog. A code is a dotted lower-case identifier; the first segment is the area. Lines about one host carry it in `CheckItem::$host` (`"host"` in the JSON), the global lines have `null` there. ## Core (`Check\Checker`) | Code | Levels | Line | |---|---|---| | `config.enabled` | warning | `enabled: false`: nothing will be submitted | | `config.dry_run` | warning, error | `dry_run` is on (error when the environment is production) | | `environment.name` | ok, warning | the `environment: …` line; warning outside production when real requests leave | | `environment.non_production_submits` | warning, error | a non-production environment with a key and `dry_run` off: error when `dry_run` was left unset, warning when it says `false` explicitly | | `config.strict_hosts` | ok, warning | `strict_hosts` on; or off next to a `hosts` map / in production | | `config.base_url` | ok, warning | `base_url` set or missing | | `config.engines` | ok | the resolved engine list | | `config.delivery` | ok | dispatch, debounce window, batch size, throttle, timeout | | `config.hosts` | error | no host to check at all (no `base_url`, no `hosts`) | | `http.client` | warning | a custom `http.client` fetches the key files: if it follows redirects, a 30x to a catch-all page looks like a 200 | | `key.missing` (host) | error | no key for the host | | `key.invalid` (host) | error | the key fails `KeyValidator` | | `key_file.location` (host) | error | `key_location` points to another host (engines answer 422) | | `key_file.served_externally` (host) | warning | `key_file.enabled: false` and no `key_location`: the web server must serve the file | | `key_file.status` (host) | ok, error | `GET /.txt`: ok on 200 with the key as body; error on any other status | | `key_file.body` (host) | error | 200 with a body that is not the key (a catch-all route) | | `key_file.fetch` (host) | error | the key file could not be fetched (network error, no HTTP client) | | `key_file.content_type` (host) | ok, warning, error | after a matching key file: `text/plain` ok; no `Content-Type` header warning; another type error; one neutral ok line when the transport exposes no headers | | `key_file.cache_control` (host) | ok, warning | after a matching key file: `Cache-Control` lifetime (`s-maxage`, else `max-age`) or `Age` above `key_file.cache_max_age` is a warning (a rotation would serve the old key for that long); absent header: no line | | `key_file.robots` (host) | ok, warning | `robots.txt` (when it answers 200): a `Disallow` covering the key file path for every bot or an engine's bot is a warning | | `key_file.previous` (host) | ok, warning | `previous_key` set: the old key file still answers 200 with the old key (ok: rotation window open), or not (warning) | | `probe.config` (host) | error | `--live`: the live configuration cannot be built | | `probe.response` (host) | ok, warning, error | `--live`: one line per engine: 200 ok, 202 warning (verification pending), anything else error | | `check.failed` | error | a registered `CheckInterface` threw; the line names the class | | `debounce.store` | ok, warning, error | `Check\DebounceStoreCheck`: off, `none`, `memory` (warning), a shared store probed ok, or unusable (error) | | `.installed` | ok, warning | `Adapter\OptionalPackage`: an optional package of the family is not installed (`sitemap.installed`); warning when its block is configured and ignored | ## Adapters | Code | Package | Levels | Line | |---|---|---|---| | `wiring.messenger` | symfony-bundle | warning | `dispatch: messenger` without a routed transport | | `wiring.doctrine` | symfony-bundle | ok, warning | entity hooks active or not | | `queue.dispatch` | laravel, yii2 | ok | `dispatch` is not `queue`: what happens instead | | `queue.connection` | laravel | error | the queue connection is not defined | | `queue.component` | yii2 | error | the yii2-queue component does not exist | | `queue.driver` | laravel, yii2 | ok, warning | the queue driver: `sync` (warning, nothing is retried) or a real one | | `eloquent.enabled` | laravel | ok, warning | model observers active or not | | `active_record.enabled` | yii2 | ok, warning | ActiveRecord hooks active or not | | `url_manager.key_file` | yii2 | ok, error | the key file is not served by the application, or `key_file` is misconfigured | | `url_manager.pretty_url` | yii2 | error | `enablePrettyUrl` is off, `/.txt` cannot be routed | | `url_manager.rule` | yii2 | ok, error | the key file URL rule is registered, or missing (component not in `bootstrap`) | | `sitemap.spool` | sitemap | ok, warning, error | where sitemap documents are spooled; error when `spool: disk` has no writable directory | Application checks (`CheckInterface` implementations you register) choose their own codes; leave the core areas (`config`, `environment`, `key`, `key_file`, `probe`, `debounce`) to the core. A line without a code is allowed but appears as `"code": null` in the JSON. # Submission store `Submission\SubmissionStoreInterface` is where the `Submitter` remembers what it did: one record per `Result` after every `submit()`, written after the listeners and the PSR-14 event. The core ships the interface, the value object `Submission\SubmissionRecord` (`urls`, `result`, `at`) and `Submission\NullSubmissionStore`, which keeps nothing and is what every adapter wires by default. The `indexnowkit/history` package (after core 0.8) brings a PSR-16 ring buffer, a PDO table, the `history` command and the `status` line; until then, implement the interface yourself or wait. ## Wiring | Where | How | |---|---| | plain PHP | `IndexNowKit::create($config, submissionStore: $store)` or `new Submitter(..., store: $store, clock: $clock)` | | `Adapter\ServicesBuilder` | `->submissionStore($store)` (an instance or a `Closure(Services): SubmissionStoreInterface`) | | Symfony bundle | replace the service `indexnowkit.submission_store` (alias `Submission\SubmissionStoreInterface`) | | Laravel | `$this->app->singleton(SubmissionStoreInterface::class, MyStore::class)` after the provider | | Yii2 | component property `submissionStore` (instance, class name, configuration array or component id) | The console submitters (`submit --force`, `--dry-run`, the sitemap command) record through the same store. ## What becomes a record | Situation | Records | |---|---| | one URL, `engines: ['api']`, 200 | 1 record, `status: ok`, `engine: api` | | one URL, `engines: ['api', 'yandex']` | 2 records, one per engine; `lastFor($url)` returns the later one, whatever its status | | 10 000 + 1 URLs, one engine | 2 records (one per batch of `batch.max_urls`) | | a URL of another host next to a URL of `base_url` | 2 records (one per host) | | `dry_run: true` | 1 record per engine × batch, `status: skipped`, `reason: dry_run`, the engine it would have reached | | `enabled: false`, a debounced URL, an unmanaged host, an invalid URL | 1 record per host (per URL for an invalid one), `status: skipped`, `engine: none` (`Result::NO_ENGINE`) | | 429 / 5xx / network failure | 1 record, `status: failed`, `retryable: true`; the retry of a queue job writes its own record later | | a listener throws | nothing changes: listeners are called before the store and are isolated | Every record gets the same `at`: the Submitter's clock (`Psr\Clock\ClockInterface`, `Clock\SystemClock` by default) read once per `submit()`. ## Contract for an implementation - `record()` must not throw. If it does, the Submitter logs `indexnow: submission store failed, {count} result(s) not recorded: {error}` once for the call and delivery is not affected. - `recent()` returns the newest records first; `host` and `status` are filters, both optional. - `lastFor($url)` matches the URL as it is stored in `Result::$urls`: normalized for everything that reached the pipeline, as given for an invalid URL (`reason: invalid_url`). An index from URL to record is the store's job; a linear scan is fine for a ring buffer of a few hundred entries. - Tier Implement ([bc.md](bc.md)): the core calls you, methods are not added in a minor. Before 1.0 the interface is new and may still move; it has to live through one minor unchanged before 1.0 is tagged. # IndexNow console runners — `indexnowkit/console` The bodies of the `check`, `submit`, `submit-`, `explain` and `key:generate` commands every framework adapter of the family ships (`bin/console indexnow:check`, `php artisan indexnow:check`, `php yii indexnow/check`), and the one declaration of their arguments and options. An adapter's command is input parsing over a runner from this package; every framework prints the same thing, and an application reuses a runner from its own command (a tenant loop over `SubmitSubjectsRunner` is a ten-line command). Split out of [`indexnowkit/core`](../core/index.md) in core 0.7 so the core no longer imports `symfony/console`; the FQCN (`IndexNowKit\Console\*`) are unchanged. [](https://packagist.org/packages/indexnowkit/console) [](https://packagist.org/packages/indexnowkit/console) [](https://github.com/indexnowkit/php/actions)   [](https://github.com/indexnowkit/php/blob/main/packages/console/LICENSE) [Русская версия](https://github.com/indexnowkit/php/blob/main/packages/console/README.ru.md) · Issues and pull requests: [github.com/indexnowkit/php](https://github.com/indexnowkit/php/issues) (the `php-*` repositories are read-only splits) ## Install ```bash composer require indexnowkit/console # brings indexnowkit/core and symfony/console ^6.4 || ^7.0 || ^8.0 ``` With a framework adapter you install nothing: `indexnowkit/symfony-bundle`, `indexnowkit/laravel` and `indexnowkit/yii2` require this package and register the commands. `indexnowkit/sitemap` builds its `sitemap` command on it too. ## What is inside | Command | Runner | What the adapter supplies | |---|---|---| | `check` | `Console\CheckRunner` | a closure that builds `Config` from the raw configuration (throws `ConfigurationException`); `Check\CheckInterface` services for adapter wiring and the add-on packages | | `submit ...` | `Console\SubmitRunner` | — | | `submit- [ids]` | `Console\SubmitSubjectsRunner` + `SubmitSubjectsOptions` | a `Console\SubjectLoaderInterface`: class resolution (FQCN or the framework's short name), objects by id, first N objects | | `explain ` | `Console\ExplainRunner` | the same loader | | `key:generate` | `Console\KeyGenerateRunner` | the default env file path | Every runner renders to a `Symfony\Component\Console\Style\SymfonyStyle` (Laravel's `OutputStyle` is one) and returns a `Console\ExitCode`. Shared by all of them: `Console\Definitions` (the arguments and options of every command, declared once — `CommandDefinition`, `ArgumentDefinition`, `OptionDefinition` — and rendered by the adapter into its framework's command), `Console\ResultFormatterInterface` (`ResultRenderer`: the table or `--json`; an application replaces it to match its own CLI), `Console\Vocabulary` (the words that differ between frameworks: "entity" / "model" / "record", `bin/console` / `php artisan` / `php yii`, where the configuration lives), `Console\ClassNameResolver` (a short class name to a FQCN, with the error texts). The submitters the commands use for `--force` / `--dry-run` (`Adapter\SubmitterFactory`) and the aggregate of a batched run (`Submission\ResultSummary`) stay in the core: they are not CLI concerns. ## Plain PHP ```php use IndexNowKit\Check\Checker; use IndexNowKit\Config; use IndexNowKit\Console\CheckRunner; use IndexNowKit\Console\Vocabulary; use IndexNowKit\IndexNowKit; use Symfony\Component\Console\Input\ArrayInput; use Symfony\Component\Console\Output\ConsoleOutput; use Symfony\Component\Console\Style\SymfonyStyle; $indexNow = IndexNowKit::create(Config::fromEnv()); $io = new SymfonyStyle(new ArrayInput([]), new ConsoleOutput()); $runner = new CheckRunner(new Checker($indexNow->config, $indexNow->keys, $indexNow->transport), new Vocabulary()); exit($runner->run($io, static fn(): Config => Config::fromEnv(), live: true)); ``` Writing an adapter? [core/docs/adapters.md §14](../core/adapters.md) walks through the six commands; the bundle, the Laravel package and the Yii2 component are the reference wirings. ## Requirements PHP 8.2+, `indexnowkit/core ^0.7`, `symfony/console ^6.4 || ^7.0 || ^8.0`. ## Notes for AI assistants - Composer package `indexnowkit/console`: the command bodies (`IndexNowKit\Console\*Runner`) and the command definitions (`IndexNowKit\Console\Definitions`) the framework adapters build their `check`, `submit`, `submit-entity` / `submit-model` / `submit-record`, `explain` and `key:generate` commands on. Framework users install an adapter, not this package. - Minimal complete snippet (every `use` included) — an application command over a runner: ```php use IndexNowKit\Console\SubmitRunner; use IndexNowKit\IndexNowKit; use Symfony\Component\Console\Style\SymfonyStyle; final class ReannounceCommand { public function __construct(private SubmitRunner $runner, private IndexNowKit $indexNow) {} public function run(SymfonyStyle $io): int { return $this->runner->run($io, ['https://www.example.com/pricing'], force: true, dryRun: false, json: false); } } ``` - Verify: the adapter's `check` command (`bin/console indexnow:check`, `php artisan indexnow:check`, `php yii indexnow/check`) is `CheckRunner`; every runner returns an `ExitCode` (`SUCCESS` 0, `FAILURE` 1, `INVALID` 2 for bad input) and never throws for remote errors. - Pitfalls: - Before core 0.7 these classes lived in `indexnowkit/core` with the same FQCN; only `Console\SubmitterFactory` (now `IndexNowKit\Adapter\SubmitterFactory`) and `Console\ResultSummary` (now `IndexNowKit\Submission\ResultSummary`) changed their namespace. - Option and argument names come from `Definitions` (`--force`, `--dry-run`, `--json`, `--live`, `--host`, `--probe-url`, `--limit`, `--event`, `--write-env`, `--length`): an adapter's command must not declare its own copies. - `--force` re-announces URLs inside the debounce window; `--dry-run` logs the request instead of sending it (`dry_run` in the configuration does the same for every submission). - Manual submission is `submitEntity()` in Symfony, `submitModel()` in Laravel, `submitRecord()` in Yii2; the commands are `indexnow:submit-entity`, `indexnow:submit-model`, `indexnow/submit-record`. - `dispatch: auto` exists in Symfony (`auto` | `messenger` | `sync` | `none`) and Yii2 (`auto` | `queue` | `sync` | `none`), **not** in Laravel (`queue` | `sync` | `none`). ## Versioning SemVer; until 1.0 minor versions may contain breaking changes, listed in [CHANGELOG.md](https://github.com/indexnowkit/php/blob/main/packages/console/CHANGELOG.md). What the compatibility promise covers: [docs/bc.md](bc.md). MIT. IndexNow is a trademark of its owner; this project is independent and not affiliated with Microsoft, Yandex or indexnow.org. # Backward compatibility `indexnowkit/console` follows SemVer and the tiers of the core's [docs/bc.md](../core/bc.md). **Before 1.0, minor versions may contain breaking changes**, listed under "Changed" in [CHANGELOG.md](https://github.com/indexnowkit/php/blob/main/packages/console/CHANGELOG.md). | Tier | Members | |---|---| | **Call** — signatures only grow by appended, defaulted parameters; pass anything past the first argument by name | `CheckRunner`, `ConfigRunner`, `SubmitRunner`, `SubmitSubjectsRunner`, `ExplainRunner`, `KeyGenerateRunner` (constructors and `run()`), `ResultRenderer`, `Vocabulary` (constructor: named arguments), `ClassNameResolver`, `Definitions::*` | | **Implement** — methods are not added without a major version | `SubjectLoaderInterface`, `ResultFormatterInterface` | | **Value objects** — `final readonly`, properties only appended with defaults | `CommandDefinition`, `ArgumentDefinition`, `OptionDefinition`, `SubmitSubjectsOptions` | | **Constants** — referenced, not hard-coded | `ExitCode::SUCCESS`, `FAILURE`, `INVALID`, `OptionDefinition::FLAG`, `VALUE`, `OPTIONAL_VALUE`, `LIST`, `CheckRunner::CONFIG_INVALID` | | **Documents** — the shape only grows by optional members | `docs/check.schema.json`, the JSON of `check --json` (`status`, `environment`, `items[].{level, code, message, host}`); the codes are the core's `docs/check-codes.md` | **Command surface.** The argument and option names, defaults and descriptions in `Definitions` are what the adapters render into their commands, so they are the public API of every adapter's CLI: an option is renamed only with a deprecation window on the adapter side. Descriptions and the printed texts of the runners are not API (they are written for humans and get improved); exit codes are. Not covered: log and exception message texts, anything under `tests/`. The package pins `indexnowkit/core ^0.8`: the runners take the core's `Config`, `Checker`, `Adapter\SubmitterFactoryInterface` and `Submission\ResultSummary`, so a core minor that changes them ships with a `console` minor. # IndexNow test kit — `indexnowkit/testing` The test suite every part of the family shares, as a `require-dev` package: the conformance scenarios of the specification as abstract PHPUnit cases you extend against *your* wiring (C01–C22 for anything that talks to the protocol, A01–A21 for an ORM adapter), the assertions of the HTTP and command scenarios (H01–H05) so a framework test parses its own response object and asserts once, an assertion for the README section AI assistants read, and the mock IndexNow server for end-to-end runs. It is what `indexnowkit/doctrine`, `indexnowkit/symfony-bundle`, `indexnowkit/laravel` and `indexnowkit/yii2` test themselves with; an adapter for another framework starts here. The four test doubles (`FakeTransport`, `ArrayLogger`, `FrozenClock`, `RecordingDispatcher`) stay in [`indexnowkit/core`](../core/index.md) under `IndexNowKit\Testing`: they implement core interfaces and need no PHPUnit, so an application test suite gets them without this package. [](https://packagist.org/packages/indexnowkit/testing) [](https://packagist.org/packages/indexnowkit/testing) [](https://github.com/indexnowkit/php/actions)   [](https://github.com/indexnowkit/php/blob/main/packages/testing/LICENSE) [Русская версия](https://github.com/indexnowkit/php/blob/main/packages/testing/README.ru.md) · Issues and pull requests: [github.com/indexnowkit/php](https://github.com/indexnowkit/php/issues) (the `php-*` repositories are read-only splits) ## Install ```bash composer require --dev indexnowkit/testing # brings indexnowkit/core; PHPUnit 11 is expected in your require-dev ``` Everything lives under `IndexNowKit\Testing\Conformance\`. ## Conformance kits Two abstract test cases turn [docs/spec/03](https://github.com/indexnowkit/spec/blob/main/03-conformance.md) into runnable scenarios against the facade your container built: ```php use IndexNowKit\IndexNowKit; use IndexNowKit\Testing\Conformance\CoreConformanceTestCase; use IndexNowKit\Testing\FakeTransport; final class CoreConformanceTest extends CoreConformanceTestCase { protected function kit(): IndexNowKit { return $this->container()->get(IndexNowKit::class); } protected function transport(): FakeTransport { return $this->container()->get(FakeTransport::class); } protected function secondHost(): ?string { return 'example.de'; } // a second entry of `hosts`, or null to skip C04 } ``` - `CoreConformanceTestCase` (C01, C03, C04, C06, C09–C12, C14, C19, C20): return the facade and the `FakeTransport` it is wired to; the scenarios use fresh URLs, so the debounce window is irrelevant. - `OrmConformanceTestCase` (A01–A21, plus A05b/A05c): implement the driver — the transaction verbs of your data layer (`begin()`, `commit()`, `rollback()`), the end of a unit of work (`flush()`, `collectedCount()`), and fixtures with fixed rule shapes (`createPost()`, `createMultiPost()`, `createCategorizedPost()`, `createTag()`, `attachTag()`, `bulkUpdateTitle()`, …). The docblock of the class lists the rules every fixture must carry; the URL conventions (`postUrl()`, `ampUrl()`, `categoryUrl()`, `homeUrl()`) are overridable. `indexnowkit/doctrine` (`tests/OrmConformanceTest.php`) and `indexnowkit/laravel` (`tests/Conformance/`) are the reference drivers. A scenario that does not apply to your framework is documented in your README, not skipped silently. The scenario identifiers are a cross-language contract: a scenario is added, never renumbered. ## Assertions for HTTP and command tests The scenarios H01–H05 are the same in every framework, only the way a response or a command output is captured differs. Parse your framework's objects, assert here: ```php use IndexNowKit\Testing\Conformance\CheckOutputAssertions; use IndexNowKit\Testing\Conformance\KeyFileAssertions; // H01: 200, text/plain, the key as the body, Cache-Control with public and max-age, Vary: Host only with a hosts map KeyFileAssertions::assertKeyFileResponse($response->getStatusCode(), $response->headers->all(), $response->getContent(), $key, maxAge: 300, expectVaryHost: true); // H02/H03: an unknown key, another host's key, key_file.enabled: false KeyFileAssertions::assertNotServed($response->getStatusCode()); // H04/H05: the check command CheckOutputAssertions::assertExitCode(0, $exitCode, $output); // the output is the failure message CheckOutputAssertions::assertReady($output, 'www.example.com'); // ": key file OK" and the closing line CheckOutputAssertions::assertKeyFileHint($output, 403); // the status and the hint about what the engines do ``` `Cache-Control` is compared by directive (frameworks order them differently), header names in any case, values as a string or a list. The phrases are the ones the core's `Checker` and the `check` command print, so your test does not carry a copy of them. `ReadmeAssertions::assertAiNotes($packageDir, $commands, $optionKeys)` checks the "Notes for AI assistants" section of a package README (EN and RU): present, with a PHP snippet that carries its `use` lines, naming only commands of the family and configuration keys the package accepts. Every package of the family runs it; an adapter of yours can too. ## The mock IndexNow server For end-to-end runs through a real PSR-18 client, without touching the engines: ```bash php -S 127.0.0.1:8089 vendor/indexnowkit/testing/resources/mock-server/router.php ``` Point `engines` at `http://127.0.0.1:8089/indexnow` (plain HTTP is accepted on loopback hosts only) and pick the behaviour with the `X-Mock-Scenario` header or `?scenario=`: `ok200` (default), `pending202`, `bad400`, `forbidden403`, `unprocessable422`, `ratelimit429` (`Retry-After: 2`), `ratelimit429-then-ok` and `flaky500-then-ok` (`?n=` failures first), `timeout`. The server validates the body like the real endpoint (host, key, `urlList`, at most 10 000 URLs, every URL on the declared host → 422 otherwise), serves `GET /{key}.txt` for the keys listed in the `MOCK_KEYS` environment variable (comma separated), and logs every request: `GET /_mock/requests` returns the log as JSON, `DELETE /_mock/requests` clears it. Start it from a test with `proc_open` on a free port, as the core's `Psr18TransportTest` does. ## Requirements PHP 8.2+, `indexnowkit/core ^0.7`, PHPUnit 11 in your `require-dev` (the test cases extend `PHPUnit\Framework\TestCase`). ## Notes for AI assistants - Composer package `indexnowkit/testing`, `require-dev` only: conformance test cases and assertions for a test suite that uses `indexnowkit/core` or one of its adapters; nothing here runs in an application. - Minimal complete snippet (every `use` included) — an adapter's conformance test: ```php use IndexNowKit\IndexNowKit; use IndexNowKit\Testing\Conformance\CoreConformanceTestCase; use IndexNowKit\Testing\FakeTransport; final class CoreConformanceTest extends CoreConformanceTestCase { protected function kit(): IndexNowKit { return $this->app->get(IndexNowKit::class); } // the facade the container built protected function transport(): FakeTransport { return $this->app->get(FakeTransport::class); } // the transport it is wired to } ``` - Verify: `vendor/bin/phpunit` runs the scenarios; a red C-scenario is a wiring problem in the adapter, not in the kit. - Pitfalls: - The test doubles (`FakeTransport`, `ArrayLogger`, `FrozenClock`, `RecordingDispatcher`) are `IndexNowKit\Testing\*` in the core; the kits and assertions are `IndexNowKit\Testing\Conformance\*` here. Before core 0.7 the assertions lived in the core under `IndexNowKit\Testing\*`. - `assertKeyFileResponse()` expects `Vary: Host` only when the application serves several hosts (a `hosts` map) and refuses it otherwise. - `CheckOutputAssertions::assertExitCode()` takes the whole output as its third argument so a failing test shows what the command printed. - The mock server accepts plain HTTP only on loopback hosts; `engines` must name the full endpoint (`http://127.0.0.1:8089/indexnow`). - `dispatch: auto` exists in Symfony (`auto` | `messenger` | `sync` | `none`) and Yii2 (`auto` | `queue` | `sync` | `none`), **not** in Laravel (`queue` | `sync` | `none`): a conformance test of an adapter runs with `dispatch: sync`. ## Versioning SemVer; until 1.0 minor versions may contain breaking changes, listed in [CHANGELOG.md](https://github.com/indexnowkit/php/blob/main/packages/testing/CHANGELOG.md). What the compatibility promise covers: [docs/bc.md](bc.md). MIT. IndexNow is a trademark of its owner; this project is independent and not affiliated with Microsoft, Yandex or indexnow.org. # Backward compatibility `indexnowkit/testing` follows SemVer and the tiers of the core's [docs/bc.md](../core/bc.md). **Before 1.0, minor versions may contain breaking changes**, listed under "Changed" in [CHANGELOG.md](https://github.com/indexnowkit/php/blob/main/packages/testing/CHANGELOG.md). | Tier | Members | |---|---| | **Call** — signatures only grow by appended, defaulted parameters; pass anything past the first argument by name | `KeyFileAssertions::*`, `CheckOutputAssertions::*`, `ReadmeAssertions::*` and their constants (`SECTION_EN`, `SECTION_RU`, `FAMILY_COMMANDS`, `CROSS_ADAPTER_KEYS` grow, never shrink) | | **Extend** — the abstract driver methods of a kit only grow by appended methods with a default implementation; a scenario is only added, never removed or renumbered, in a minor | `CoreConformanceTestCase`, `OrmConformanceTestCase` | | **Resource** — the file path and the scenario names are stable; scenarios are added, not renamed | `resources/mock-server/router.php` (`X-Mock-Scenario`: `ok200`, `pending202`, `bad400`, `forbidden403`, `unprocessable422`, `ratelimit429`, `ratelimit429-then-ok`, `flaky500-then-ok`, `timeout`; `GET /_mock/requests`, `DELETE /_mock/requests`; `MOCK_KEYS`) | The conformance identifiers (C01–C22, A01–A21, H01–H06) are a cross-language contract of the specification: they are frozen at 1.0 of the family. What an assertion *accepts* may become stricter in a minor when the specification does (listed in the changelog); what it *rejects* never becomes accepted silently. Not covered: the failure-message texts of the assertions, anything under `tests/`. The package pins `indexnowkit/core ^0.7`: it reads `Config::OPTIONS` and the test doubles of the core, so a core minor that renames them ships with a `testing` minor. # IndexNow sitemap reader — `indexnowkit/sitemap` Re-announce a site's URLs to Yandex, Bing and the other [IndexNow](https://www.indexnow.org) engines from its own sitemap: a sitemap index, gzip-compressed and text sitemaps are streamed entry by entry and submitted in batches, so a million-URL sitemap never lives in memory. The `sitemap` command of every framework adapter of the family (`indexnowkit/symfony-bundle`, `laravel`, `yii2`) is this package; in plain PHP it is three lines over [`indexnowkit/core`](../core/index.md). **Google: no.** Google does not support IndexNow, its sitemap ping endpoint is gone and the Indexing API is limited to `JobPosting` / `BroadcastEvent`. Keep your sitemap for Google; this package announces it to the IndexNow engines only. IndexNow is a notification, not indexing: the engine decides whether and when to crawl. A run without `--changed-since` re-announces the whole sitemap: do that once, then schedule `--changed-since "1 day"`. `--changed-since` relies on ``; a generator that writes `lastmod = now()` for every URL turns every run into a full run, and entries without `lastmod` are skipped when the option is set. [](https://packagist.org/packages/indexnowkit/sitemap) [](https://packagist.org/packages/indexnowkit/sitemap) [](https://github.com/indexnowkit/php/actions)    [](https://github.com/indexnowkit/php/blob/main/packages/sitemap/LICENSE) [Русская версия](https://github.com/indexnowkit/php/blob/main/packages/sitemap/README.ru.md) · Issues and pull requests: [github.com/indexnowkit/php](https://github.com/indexnowkit/php/issues) (the `php-*` repositories are read-only splits) ## Install ```bash composer require indexnowkit/sitemap # brings indexnowkit/core; needs ext-xmlreader, ext-zlib for .gz ``` With a framework adapter you install nothing: the adapter requires this package and registers the command (`bin/console indexnow:sitemap`, `php artisan indexnow:sitemap`, `php yii indexnow/sitemap`) with its `sitemap` configuration block. ## Plain PHP ```php use IndexNowKit\Config; use IndexNowKit\IndexNowKit; use IndexNowKit\Sitemap\SitemapConfig; use IndexNowKit\Sitemap\SitemapReader; $kit = IndexNowKit::create(Config::fromEnv()); $reader = SitemapReader::fromConfig(SitemapConfig::fromArray(['spool' => 'auto']), $kit->transport); $batch = []; foreach ($reader->read('https://www.example.com/sitemap.xml', new DateTimeImmutable('-1 day')) as $entry) { $batch[] = $entry->url; if (\count($batch) === $kit->config->batchMaxUrls) { $kit->submit($batch); $batch = []; } } $kit->submit($batch); ``` `read()` yields `SitemapEntry` objects (`url`, `lastmod`), optionally only those whose `` is newer than `$changedSince` (entries without `lastmod` are then skipped). The root may be an http(s) URL, a local path or a `file://` URL; nested sitemaps of an index are fetched over the transport you pass (the one the facade submits through, so `http.client` and `http.timeout` apply). `$kit->transport` is `null` when the facade was built around a custom submitter: use `Http\TransportFactory::lazy($kit->config)` then. ## Configuration `SitemapConfig::fromArray()` reads the `sitemap` block every adapter exposes; `SitemapConfig::OPTIONS` lists its dotted keys for `Config::unknownOptions()`. | Key | Default | | |---|---|---| | `sitemap.enabled` | `true` | `false`: the adapter registers no command and no reader | | `sitemap.url` | `null` | sitemap read when the command gets no argument; `null` = `/sitemap.xml` | | `sitemap.max_depth` | `3` | levels of `` followed below the root (`0` = the root only) | | `sitemap.max_sitemaps` | `1000` | documents fetched per run, root included | | `sitemap.max_bytes` | `52428800` | size cap of one uncompressed document (50 MiB, the protocol maximum; at least 1024) | | `sitemap.allow_foreign_hosts` | `false` | follow nested sitemaps on other origins (CDN-hosted parts); `--allow-foreign-hosts` enables it for one run | | `sitemap.spool` | `auto` | where a document is kept while parsing: `auto` = temp file, memory when the temp dir is not writable; `disk` = temp file or fail; `memory` | | `sitemap.spool_dir` | `null` | directory of the temp files (`sys_get_temp_dir()`); point it at a writable volume on a read-only filesystem | | `sitemap.fetch_retries` | `2` | extra attempts (1 s, 2 s, 4 s apart) after a network failure or 5xx while fetching a document; 4xx and broken documents are never retried | ## How it stays safe and small Memory stays flat whatever the sitemap size: every document is spooled (`Sitemap\Spool`: a temp file, or memory on a read-only filesystem; straight from the socket when the transport implements `Http\StreamingTransportInterface`, as `Psr18Transport` does), gzip is inflated chunk by chunk into a second spool, and `XMLReader` walks the spool through the `indexnowkit-spool://` wrapper with a few KiB of buffers. Nested sitemaps must live on the origin of the root unless `allow_foreign_hosts` says otherwise; recursion depth, document count and document size (before and after gunzip) are capped; external entities and network access are disabled in the XML parser. A failing nested sitemap is logged and skipped; a failing root throws `Http\Exception\TransportException`, and a response shorter than its `Content-Length` is a truncated download, never a document. Details in [SECURITY.md](https://github.com/indexnowkit/php/blob/main/packages/sitemap/SECURITY.md). ## The command `Sitemap\Console\SitemapRunner` is the body of `sitemap [url]` (`--changed-since "1 day"`, `--allow-foreign-hosts`, `--force`, `--dry-run`, `--json`); it streams, submits every `batch.max_urls` URLs, and submits the pending batch before reporting a mid-run failure (the re-run is idempotent, what was read is still worth announcing). `--force` ignores the debounce window (URLs announced within the last `debounce.per_url` seconds are sent again): for a deliberate one-off re-run, never in a schedule. `batch.max_urls` (10 000) is the protocol's ceiling, not a target: smaller batches are accepted just as well, and a scheduled run with `--changed-since` normally sends a few. The `check` command of every adapter carries `Sitemap\Check\SitemapSpoolCheck`: where documents are spooled, and whether that directory is writable — the kind of thing that otherwise only shows up on the first scheduled run. An application decorates the source (filter, rewrite) or replaces it (another format, a database) by implementing `Sitemap\SitemapSourceInterface` and binding it under the adapter's alias. Writing an adapter? [docs/adapters.md](adapters.md). ## Requirements PHP 8.2+, `ext-xmlreader`, `indexnowkit/core ^0.7`, `indexnowkit/console ^0.1` (the command body renders through `symfony/console`); `ext-zlib` for gzip-compressed sitemaps. ## Notes for AI assistants - Composer package `indexnowkit/sitemap` (add-on of `indexnowkit/core`; `ext-xmlreader`, `ext-zlib` for gzip). With a framework adapter it is the `sitemap` command — nothing to wire. - Minimal complete snippet (every `use` included): ```php use IndexNowKit\Config; use IndexNowKit\IndexNowKit; use IndexNowKit\Sitemap\SitemapConfig; use IndexNowKit\Sitemap\SitemapReader; $indexNow = IndexNowKit::create(Config::fromEnv()); $reader = SitemapReader::fromConfig(SitemapConfig::fromArray(['spool' => 'auto']), $indexNow->transport); foreach ($reader->read('https://www.example.com/sitemap.xml') as $entry) { $indexNow->collect([$entry->url]); } $indexNow->flush(); // batches of batch.max_urls, debounced ``` - Verify: the adapter's `check` command prints the `sitemap:` spool line; `bin/console indexnow:sitemap --dry-run`, `php artisan indexnow:sitemap --dry-run`, `php yii indexnow/sitemap --dry-run`. - Pitfalls: - `dispatch: auto` exists in Symfony (`auto` | `messenger` | `sync` | `none`) and Yii2 (`auto` | `queue` | `sync` | `none`), **not** in Laravel (`queue` | `sync` | `none`). - Locales: `router.locales` in Laravel, `router.languages` in Yii2, `framework.enabled_locales` in Symfony; `locales: 'all'` on a rule uses that list. - `url:` names an accessor (method or property) that returns the URL; `urls:` is a list of literal URLs. Never put a literal in `url:`. - A string in `when:` is an accessor read as truthy (`published`, `isPublished`). A status string needs `Equals`: `when: new Equals('status', 'published')` (`IndexNowKit\Attribute\Param\Equals`). - Manual submission is `submitEntity()` in Symfony, `submitModel()` in Laravel, `submitRecord()` in Yii2; the commands are `indexnow:submit-entity`, `indexnow:submit-model`, `indexnow/submit-record`. Bulk queries (`update()`, `DB::table()`, `updateAll()`) fire no hooks: submit afterwards with those. - Laravel has two classes called `IndexNowKit`: the facade `IndexNowKit\Laravel\Facades\IndexNowKit` and the core service `IndexNowKit\IndexNowKit` (inject by type). Yii2 exposes the core through `Yii::$app->indexnow->kit()`. - Outside production a configured key with `dry_run` unset makes `check` fail (a staging copy would submit real URLs): set `dry_run: true` there, or `dry_run: false` explicitly when it submits on purpose. - Unknown configuration keys are warned about at boot (typos such as debounce.per_urls); the key list is `Config::OPTIONS` plus the adapter's own keys. ## Versioning SemVer; until 1.0 minor versions may contain breaking changes, listed in [CHANGELOG.md](https://github.com/indexnowkit/php/blob/main/packages/sitemap/CHANGELOG.md). What the compatibility promise covers: [docs/bc.md](bc.md). MIT. IndexNow is a trademark of its owner; this project is independent and not affiliated with Microsoft, Yandex or indexnow.org. # Wiring the `sitemap` command into an adapter The command body, the reader and the check live here; an adapter parses its own input and binds three objects. Everything reads one `SitemapConfig`, built from the raw `sitemap` block of the adapter's configuration. ```php use IndexNowKit\Sitemap\Check\SitemapSpoolCheck; use IndexNowKit\Sitemap\Console\SitemapOptions; use IndexNowKit\Sitemap\Console\SitemapRunner; use IndexNowKit\Sitemap\SitemapConfig; use IndexNowKit\Sitemap\SitemapReader; $sitemap = SitemapConfig::fromArray($raw['sitemap'] ?? []); // throws ConfigurationException naming the key $reader = SitemapReader::fromConfig($sitemap, $kit->transport ?? TransportFactory::lazy($kit->config), $logger); $check = new SitemapSpoolCheck($sitemap); // add it to the checks of your `check` command $runner = new SitemapRunner($kit, $reader, $submitterFactory, $sitemap->url, $formatter, sitemapUrlOption: 'myfw.sitemap.url'); $exit = $runner->run($io, new SitemapOptions($argument, $changedSince, $allowForeignHosts, $force, $dryRun, $json)); ``` - **Configuration.** Add `SitemapConfig::OPTIONS` to the keys your `Adapter\ConfigFactory` owns (`ownedOptions: [...MY_OPTIONS, ...SitemapConfig::OPTIONS]`), so a typo inside the block is warned about. Do not list a bare `sitemap` key: it would stop `Config::unknownOptions()` from looking inside the block. At runtime build the block with `SitemapConfig::loadOrDisabled($block, $logger, 'php artisan indexnow:check')`: an invalid block is one `critical` line and `SitemapConfig::disabled()`, the way the core `ConfigFactory::load()` treats the core options (a DI container that validates at compile time, like the bundle, uses `fromArray()` and lets the build fail); when `enabled` is false, register no command (bundle) or refuse to run it with `sitemap.enabled is false.` and exit `INVALID` (Laravel, Yii2). - **Log lines.** The one line the wiring above adds, so operators can grep for it (the core's `docs/operations.md` lists the rest): | Level | Message | |---|---| | `critical` | `indexnow: invalid sitemap configuration, the sitemap command is disabled until it is fixed: {error} (run "{check}")` — `loadOrDisabled()`, `{check}` is the adapter's check command (Laravel, Yii2) | - **Transport.** The reader fetches over the transport the facade submits through, so `http.client` and `http.timeout` apply and nothing is discovered twice. `$kit->transport` is `null` only when the facade was built around a custom submitter; `Http\TransportFactory::lazy($kit->config)` covers that. - **Source.** Type the command against `SitemapSourceInterface` and expose the reader under an alias of it, so an application can decorate the source (filter, rewrite) or replace it. `--allow-foreign-hosts` only reaches the shipped `SitemapReader`; the runner warns when the configured source is something else. - **Output.** The runner streams, submits every `batch.max_urls` URLs through `Adapter\SubmitterFactory::choose()` (`--force`/`--dry-run` get a separate submitter), folds results into `Submission\ResultSummary`, and submits the pending batch before reporting a mid-run failure; `--json` keeps stdout machine-readable (the error goes to stderr). Exit codes are `Console\ExitCode` of `indexnowkit/console`. - **Words.** The only framework-specific string is `sitemapUrlOption`, printed in `Give a sitemap URL, or configure or base_url.` when no sitemap is known. The reference wiring: `IndexNowKitLoader` of the Symfony bundle (services `indexnowkit.sitemap_config`, `indexnowkit.sitemap_reader`, `indexnowkit.console.sitemap`), `IndexNowKitServiceProvider::registerDiagnostics()` in Laravel, `IndexNowComponent::sitemapConfig()` / `sitemapSource()` in Yii2. # Backward compatibility `indexnowkit/sitemap` follows SemVer and the tiers of the core's [docs/bc.md](../core/bc.md). **Before 1.0, minor versions may contain breaking changes**, listed under "Changed" in [CHANGELOG.md](https://github.com/indexnowkit/php/blob/main/packages/sitemap/CHANGELOG.md). | Tier | Members | |---|---| | **Call** — signatures only grow by appended, defaulted parameters; pass anything past the first argument by name | `SitemapReader` (constructor, `fromConfig()`, `read()`, `parse()`), `SitemapConfig` (constructor, `fromArray()`, `disabled()`), `Console\SitemapRunner`, `Console\SitemapOptions`, `Check\SitemapSpoolCheck`, `Spool::create()`, `probeDisk()`, `uri()`, `close()` | | **Implement** — methods are not added without a major version | `SitemapSourceInterface` | | **Value objects** — `final readonly`, properties only appended with defaults | `SitemapEntry`, `SitemapConfig` | | **Constants** — referenced, not hard-coded; values may change in a minor | `SitemapReader::MAX_XML_BYTES`, `MAX_SITEMAPS`, `SitemapConfig::OPTIONS`, `DEFAULT_MAX_DEPTH`, `DEFAULT_FETCH_RETRIES`, `MIN_MAX_BYTES`, `Spool::SCHEME` | | **Enum** — closed set | `SpoolMode` | Not covered: `Spool`'s `stream_*` methods (the PHP stream-wrapper protocol, `@internal`), log and exception message texts, anything under `tests/`. Adapters pin `indexnowkit/sitemap`, `indexnowkit/core` and `indexnowkit/console` together: a `SitemapReader` of this package over an older core is refused by Composer, on purpose (0.1 clashed with the classes core 0.3 still shipped; 0.3 needs the `Adapter\SubmitterFactory` of core 0.7 and the runners of `indexnowkit/console`). # `indexnowkit/doctrine` — commit-safe IndexNow for Doctrine ORM Listens to `onFlush` / `postFlush`, resolves the URLs of entities that declare `#[IndexNow]` rules, and hands them over **only after the outermost transaction really committed**, using a DBAL driver middleware. Rolled-back flushes submit nothing. Deletions are resolved before the row disappears. Doctrine ORM 2.19+ and 3.x, DBAL 3.x and 4.x, PHP 8.2+. [Русская версия](https://github.com/indexnowkit/php/blob/main/packages/doctrine/README.ru.md) · Issues and pull requests: [github.com/indexnowkit/php](https://github.com/indexnowkit/php/issues) (the `php-*` repositories are read-only splits) [](https://packagist.org/packages/indexnowkit/doctrine) [](https://packagist.org/packages/indexnowkit/doctrine) [](https://github.com/indexnowkit/php/actions) [](https://github.com/indexnowkit/spec)   [](https://github.com/indexnowkit/php/blob/main/packages/doctrine/LICENSE) **Symfony users: take [`indexnowkit/symfony-bundle`](../symfony-bundle/index.md)** — it wires all of this, adds the router bridge, the commands and the profiler panel. This package is for Doctrine without Symfony. ## Why this over X Most IndexNow packages are a thin HTTP client: you collect the URLs, you call it, you read the answer. This family does the part that goes wrong in practice: - **Declared on the model** (`#[IndexNow]`) and submitted from the ORM hooks — no controller code to forget. - **After the commit**, not on flush: a rolled-back transaction announces nothing. - **Debounce** (10 minutes per URL, shared through your cache), **batches** of up to 10 000 URLs, one key per host from env. - **Answers handled**: 202 (key pending), 422, 429 with `Retry-After` back-off and a retry through your queue, 403 escalation. - **`check` and `explain`** in the Symfony bundle say what is wrong before the first submission and why a URL was or was not sent. - **One core** under the Symfony, Laravel, Yii2 and Doctrine adapters with a shared conformance suite: the same behaviour everywhere, documented once. ## Install ```bash composer require indexnowkit/doctrine ``` ## Standalone wiring ```php use Doctrine\DBAL\DriverManager; use Doctrine\ORM\{EntityManager, ORMSetup}; use IndexNowKit\{Config, IndexNowKit}; use IndexNowKit\Doctrine\IndexNowDoctrine; use IndexNowKit\Url\{ArrayResolverLocator, AttributeUrlResolver}; $indexNow = IndexNowKit::create(Config::fromEnv(), logger: $logger); $resolver = new AttributeUrlResolver( $indexNow->attributes, router: null, // no framework router: see "Routes" below locator: new ArrayResolverLocator([ 'post_url' => fn (Post $post): string => '/posts/' . $post->slug, // #[IndexNow(resolver: 'post_url')] ]), logger: $logger, ); $wiring = new IndexNowDoctrine($indexNow, $resolver, $logger, autoFlush: true); $ormConfiguration = ORMSetup::createAttributeMetadataConfiguration([__DIR__ . '/src/Entity'], isDevMode: false); $wiring->registerMiddleware($ormConfiguration); // BEFORE DriverManager::getConnection() $connection = DriverManager::getConnection(['driver' => 'pdo_sqlite', 'path' => __DIR__ . '/var/app.db'], $ormConfiguration); $entityManager = new EntityManager($connection, $ormConfiguration); $wiring->registerListener($entityManager); ``` `registerMiddleware()` must run before the connection is created, because DBAL middlewares wrap the driver at connect time. In a typical bootstrap that means: build the ORM `Configuration`, call `registerMiddleware()`, then create the `EntityManager`, then call `registerListener()`. `$autoFlush: true` submits as soon as the URLs are handed over, which is what a script or a CLI process wants. Pass `false` and call `$indexNow->flush()` yourself at the end of the unit of work when you control the request cycle. `IndexNowDoctrine` exposes the three pieces it builds — `$wiring->staging`, `$wiring->listener`, `$wiring->middleware` — so a container can register them individually instead. ## Declaring pages The `#[IndexNow]` attribute comes from the core and is repeatable: one rule per family of public URLs. ```php use IndexNowKit\Attribute\{IndexNow, IndexNowDefaults}; #[ORM\Entity] #[IndexNowDefaults(when: 'isPublished', fields: ['slug', 'title', 'body', 'published'])] #[IndexNow(resolver: 'post_url')] #[IndexNow(via: 'category')] // a changed post also refreshes its category page #[IndexNow(urls: ['/'])] // and the homepage class Post { #[ORM\Column] private bool $published = false; public function isPublished(): bool { return $this->published; } } ``` Full model — sources, typed parameters, `when` / `whenFields` / `fields` / `events` / `locales` / `host`, inheritance and the semantics table — is in the core's [attribute reference](../core/attribute-reference.md). ### Routes `#[IndexNow(route: ...)]` needs a `RouteUrlResolverInterface` bridge to a framework router. Standalone Doctrine has none, so a rule using `route:` fails at resolution time with *"no router bridge is configured"* (logged, never thrown into your flush). Use `url:`, `urls:` or `resolver:` instead, or implement the two-method `RouteUrlResolverInterface` for your own router and pass it as the `router:` argument above. ## What the listener does In `onFlush`, every scheduled insertion, update, deletion and changed collection is classified **per rule** through the core's `ObjectChangeHandler`: - **Insertions** produce `created` events. Their URLs are resolved in `postFlush`, once identifiers are assigned. - **Updates** are classified per rule from `UnitOfWork::getEntityChangeSet()`. A rule whose `when` turned false becomes a **deletion** and is resolved immediately in `onFlush`, while the old state is still live; a rule whose `when` turned true becomes a creation; otherwise it is an update, filtered by the rule's `fields`. One entity can therefore produce an update for one page and a deletion for another in the same flush. - **Changed to-many associations** are not part of the owner's change set, so a scheduled collection update or deletion re-classifies its owner with the association's field name as the changed field. Changing `post.tags` resubmits the post's pages. - **Deletions** are resolved in `onFlush`, before the row disappears. A rule that does not apply — a draft that was never public — submits nothing. In `postFlush` the deferred rules are resolved, every URL is logged at `debug` with the rule that produced it (`indexnow: App\Entity\Post#post_amp (updated) -> https://example.com/amp/hello`), and the batch is handed off. Nothing here throws into your application. An invalid attribute, an unreadable `when` accessor or a failing resolver is logged on the `indexnow` channel and yields no URLs. ## Renamed pages When a field a route parameter reads changes — the slug, the category the path goes through — the old URL now answers 404. On an update the listener resolves the rule against the **previous** values of the change set and announces those URLs as deleted, next to the new URLs as updated, in the same flush (`ObjectChangeHandler::renamed()`, scenario A21). Route rules only; the old page must have been public (`when` true before the change); a field the URL depends on that cannot be written back (`readonly`, uninitialized) skips the old URL with a `debug` line. Nothing in this path throws into `flush()`. ## Commit safety `postFlush` runs before the outer `COMMIT` whenever `flush()` is wrapped in `wrapInTransaction()` or a manual transaction, and Doctrine has no after-commit event. So: - if the connection has an open transaction, the URLs are staged against its **native** connection object; - the DBAL driver middleware sees the real `commit()` and `rollBack()` — nesting level 0, identically in DBAL 3 and 4 (`Middleware\IndexNowConnection` / `IndexNowConnectionV3`, picked by `IndexNowDriver` at connect time) — and either releases the staged URLs or discards them; - a `commit()` that itself throws discards them too, so a pooled connection never delivers them later; - a nested transaction rolled back to its savepoint (`ROLLBACK TO SAVEPOINT`, what DBAL issues for an inner `rollBack()`) drops the URLs staged inside it; the outer `COMMIT` delivers the rest; - outside a transaction the URLs are handed over immediately. If the driver exposes no native connection object, the listener logs a warning and submits inside the open transaction rather than losing the URLs. ## Limitations - DQL and QueryBuilder bulk `UPDATE` / `DELETE`, and `Connection::executeStatement()`, bypass the unit of work and are not detected. Submit those URLs with `$indexNow->submit()`. - `route:` needs a router bridge (see above). - Entities inserted through `INSERT ... SELECT` never reach `postFlush`. - Attributes are not read from interfaces or traits: PHP does not inherit class attributes through them, and Doctrine mapping behaves the same way. ## Compatibility with other listeners Register the listener **after** anything that computes values the URLs depend on. With Gedmo Sluggable the slug is written in `onFlush`, so the IndexNow listener must run later; the Symfony bundle uses priority `-100` for exactly this reason. ## Compatibility Public API of this package: the classes named in the changelog and the README, their constructor parameter names (pass optional arguments by name), and the DBAL middleware classes. The core's rules apply, including the "may grow" interfaces: [bc.md](../core/bc.md); what this package itself keeps stable: [docs/bc.md](bc.md). Before 1.0 a minor version may break; every break is listed under "Changed" in [CHANGELOG.md](https://github.com/indexnowkit/php/blob/main/packages/doctrine/CHANGELOG.md) with the migration. ## Notes for AI assistants - Composer package `indexnowkit/doctrine` (Doctrine ORM 2.19+/3, DBAL 3/4, on `indexnowkit/core`). In a Symfony application use `indexnowkit/symfony-bundle`, which wires this package. - Minimal complete snippet (every `use` included): ```php use Doctrine\ORM\Mapping as ORM; use IndexNowKit\Attribute\{IndexNow, IndexNowDefaults}; #[ORM\Entity] #[IndexNowDefaults(when: 'isPublished')] #[IndexNow(route: 'post_show', params: ['slug' => 'slug'])] // a Symfony route, or resolver: 'post_url' without a router class Post { /* ORM columns, isPublished() */ } ``` - Verify: with the bundle, `bin/console indexnow:check` and `bin/console indexnow:explain 'App\\Entity\\Post' 1`; standalone, `$indexNow->explain($post)` returns the resolved URLs with the rule that produced each. - Pitfalls: - `dispatch: auto` exists in Symfony (`auto` | `messenger` | `sync` | `none`) and Yii2 (`auto` | `queue` | `sync` | `none`), **not** in Laravel (`queue` | `sync` | `none`). - Locales: `router.locales` in Laravel, `router.languages` in Yii2, `framework.enabled_locales` in Symfony; `locales: 'all'` on a rule uses that list. - `url:` names an accessor (method or property) that returns the URL; `urls:` is a list of literal URLs. Never put a literal in `url:`. - A string in `when:` is an accessor read as truthy (`published`, `isPublished`). A status string needs `Equals`: `when: new Equals('status', 'published')` (`IndexNowKit\Attribute\Param\Equals`). - Manual submission is `submitEntity()` in Symfony, `submitModel()` in Laravel, `submitRecord()` in Yii2; the commands are `indexnow:submit-entity`, `indexnow:submit-model`, `indexnow/submit-record`. Bulk queries (`update()`, `DB::table()`, `updateAll()`) fire no hooks: submit afterwards with those. - Laravel has two classes called `IndexNowKit`: the facade `IndexNowKit\Laravel\Facades\IndexNowKit` and the core service `IndexNowKit\IndexNowKit` (inject by type). Yii2 exposes the core through `Yii::$app->indexnow->kit()`. - Outside production a configured key with `dry_run` unset makes `check` fail (a staging copy would submit real URLs): set `dry_run: true` there, or `dry_run: false` explicitly when it submits on purpose. - Unknown configuration keys are warned about at boot (typos such as debounce.per_urls); the key list is `Config::OPTIONS` plus the adapter's own keys. ## Documentation | | | |---|---| | Attribute reference | [core/docs/attribute-reference.md](../core/attribute-reference.md) | | Configuration | [core/docs/configuration.md](../core/configuration.md) | | Operations and logging | [core/docs/operations.md](../core/operations.md) | | Testing | [core/docs/testing.md](../core/testing.md) | | Writing your own adapter | [core/docs/adapters.md](../core/adapters.md) | | Design rationale | [docs/spec](https://github.com/indexnowkit/spec) | Changelog: [CHANGELOG.md](https://github.com/indexnowkit/php/blob/main/packages/doctrine/CHANGELOG.md). Versioning: SemVer; before 1.0 minor versions may break. MIT. IndexNow is a trademark of its owner; this project is independent and not affiliated with Microsoft, Yandex or indexnow.org. # Backward compatibility `indexnowkit/doctrine` follows SemVer. **Before 1.0, minor versions may contain breaking changes**; every one is listed under "Changed" in [CHANGELOG.md](https://github.com/indexnowkit/php/blob/main/packages/doctrine/CHANGELOG.md) with the migration. After 1.0 the rules below become the promise. The core's tiers ("call", "implement", "may grow") apply to every core class you touch through this package: [core bc.md](../core/bc.md). ## What the package keeps stable | Surface | Promise | |---|---| | **`IndexNowDoctrine`** (`registerMiddleware()`, `registerListener()`, the constructor's named parameters) | Method and parameter names stay; new parameters are appended with defaults. | | **`IndexNowListener`** (the `onFlush` / `postFlush` subscriber) and **`Middleware\IndexNowMiddleware`** | Their constructors' named parameters and the Doctrine events they subscribe to stay. They are `final`: put your own logic on top of the core `ObjectChangeHandler`, not on a subclass. | | **Behaviour** described in the README: URLs resolved in `onFlush` while the old state is live, handed over only after the outermost transaction committed, nothing on rollback, deletions resolved before the row disappears, savepoints honoured | A change to any of these is a breaking change and appears in the changelog with the conformance ids (A01–A21) it affects. | | **DBAL 3 and 4, ORM 2.19+ and 3** | Dropping a major of either is a breaking change of this package. | Not a contract: `Middleware\IndexNowDriver`, `Middleware\IndexNowConnection`, `Middleware\IndexNowConnectionV3`, `Middleware\SavepointStatement` (the DBAL wrapping, which follows DBAL's own API), and log message texts (their `context` keys are). ## Pinning `composer require indexnowkit/doctrine:^0.5` gets every 0.5.x patch. Read the changelog before a minor. # Backward compatibility `indexnowkit/symfony-bundle` follows SemVer. **Before 1.0, minor versions may contain breaking changes**; every one is listed under "Changed" in [CHANGELOG.md](https://github.com/indexnowkit/php/blob/main/packages/symfony-bundle/CHANGELOG.md) with the migration. After 1.0 the rules below become the promise. The core's tiers ("call", "implement", "may grow") apply to every core class you touch through the bundle: [core bc.md](../core/bc.md). ## What the bundle keeps stable | Surface | Promise | |---|---| | **Configuration keys** of `indexnowkit:` ([configuration.md](configuration.md)) | Keys and their meaning stay; new keys are only added with a default. A rename ships the old key as deprecated for one minor (`serve_key_file` → `key_file.enabled` is the example) and is listed in the changelog. | | **Environment variables** the recipe reads (`INDEXNOW_KEY`, `INDEXNOW_BASE_URL`, …) | Names stay; new ones are only added. | | **Command names and options** (`indexnow:check`, `indexnow:key:generate`, `indexnow:submit`, `indexnow:submit-entity`, `indexnow:explain`, `indexnow:sitemap`) | Names, arguments and options come from the core `Console\Definitions`; new options are only added. Output is not a contract except the exit codes and the `--json` shape of the core formatter. | | **Service ids and aliases** listed in [extending.md](extending.md) (`indexnowkit`, `indexnowkit.config`, `indexnowkit.submitter`, `indexnowkit.transport`, the `Interface` aliases, the `indexnowkit.check` tag) | Ids and the interface each one implements stay; decorating them stays possible. Ids not in that table (`indexnowkit.check.debounce_store.probe`, `indexnowkit.console.*` internals) are implementation details. | | **Container parameters** `indexnowkit.dispatch`, `indexnowkit.messenger_routed`, `indexnowkit.doctrine_hooked` | Names and types stay. | | **Public classes** `IndexNowKitBundle`, `Command\*`, `Controller\KeyFileController`, `Routing\KeyFileRouteLoader`, `EventListener\FlushListener`, `Check\WiringCheck`, `Check\CacheProbe`, `Messenger\SubmitUrlsMessage`, `DataCollector\IndexNowDataCollector` | Constructor parameters are passed by name; new ones are appended with defaults. They are `final`: extend by decoration, not inheritance. | | **Route** `indexnowkit_key_file` (or `key_file.route_name`) | The route name and the `{key}.txt` shape stay. | Not a contract: the Twig templates of the profiler panel, log message texts (their `context` keys are), the exact wording `indexnow:check` prints (its exit code and the ok/warning/error levels are), and the compiled container shape the bundle's own tests pin. ## Pinning `composer require indexnowkit/symfony-bundle:^0.8` gets every 0.8.x patch. Read the changelog before a minor. # Configuration reference Every node of the `indexnowkit` extension, with its default and what it does. Anything that is not an environment placeholder is validated when the container is compiled, so a typo fails at `cache:clear`, not at the first submission. ```yaml # config/packages/indexnowkit.yaml indexnowkit: # Master switch. false = collect nothing, submit nothing, register no Doctrine listener. # Prefer dry_run if you still want the logs and the profiler panel. enabled: true # Default IndexNow key, 8-128 characters of [A-Za-z0-9-]. # Generate: bin/console indexnow:key:generate --write-env key: null # '%env(INDEXNOW_KEY)%' # Absolute site URL. Needed outside HTTP requests (console commands, Messenger workers) and to # resolve relative URLs. Required when dispatch is "messenger", and whenever `hosts` is used. base_url: null # '%env(INDEXNOW_BASE_URL)%' # Absolute URL of the key file when it is NOT served at https:///.txt. # Must be on the host of base_url. key_location: null # The key before a rotation: /.txt keeps being served while engines re-verify, # nothing is ever submitted under it. Remove it once the rotation settled. previous_key: null # Multi-domain: one entry per additional host. Hosts not listed here use the default key, # unless strict_hosts is on. An array node cannot come from a single env var, so use # per-entry %env(...)% placeholders. hosts: # example.de: # key: '%env(INDEXNOW_KEY_DE)%' # key_location: 'https://example.de/keys/indexnow.txt' # optional # base_url: 'https://example.de' # console/worker URL generation # engines: [yandex] # this host only; default: `engines` # previous_key: '%env(INDEXNOW_KEY_DE_OLD)%' # rotation window # Refuse URLs of hosts that are neither base_url nor listed in `hosts`, instead of announcing # them under the default key. Recommended for multi-domain setups, and for any production # app that is also reachable under a staging or internal hostname (indexnow:check warns). strict_hosts: false # Environment names treated as production: no dry-run safety net there, and indexnow:check # flags dry_run. Replace (not extend) the list when yours is not prod/production. production_environments: [prod, production] # URLs longer than this many bytes are skipped as invalid_url. The protocol sets no limit. max_url_length: 2048 # api = api.indexnow.org, the shared endpoint reaching Yandex, Bing, Naver, Seznam, Yep, Internet Archive and Amazon. # Name a single engine only to target it, or give a full https endpoint URL (or an alias below). engines: [api] # Short names for custom endpoints, usable in engines and hosts..engines. engine_aliases: {} # { corp: 'https://index.corp.example/indexnow' } # locale => host. A rule with `locales` and no `host` generates each locale on its host # (en on www.example.com, de on example.de). List those hosts in `hosts` with their keys. locale_hosts: {} # { en: www.example.com, de: example.de } # auto = messenger when a Messenger transport is configured, otherwise sync. # sync = submit after the response was sent (kernel.terminate). none = collect, never send. dispatch: auto # auto | sync | messenger | none messenger: # Bus service id. It needs the dispatch_after_current_bus middleware; default buses have it. bus: messenger.default_bus # Transport name from framework.messenger.transports. When set, the bundle adds the routing # for SubmitUrlsMessage itself, so messenger.yaml needs no edit. transport: null # e.g. 'async' # Milliseconds: a DelayStamp on every SubmitUrlsMessage. Needs a transport that supports # delays (doctrine, amqp, redis, sqs); in-memory and sync transports ignore it. delay: 0 # Service ids of extra stamps added to every message (a FIFO group id, a priority). stamps: [] batch: # URLs per request. The protocol maximum is 10000; larger sets are split. max_urls: 10000 debounce: # Seconds during which the same URL is not re-submitted. Yandex accepts the same URL at most # once per 10 minutes. 0 disables debouncing. per_url: 600 # 'memory' (per process: CLI, tests), 'none', or a PSR-6 cache pool service id shared by all # processes. store: cache.app # Cache key prefix. Give each application sharing one pool its own. key_prefix: indexnowkit_ throttle: # Outgoing requests per minute, per process: N workers get N buckets. For a site-wide limit replace # indexnowkit.throttle (ThrottleInterface) with a shared limiter, e.g. on symfony/rate-limiter + Redis. max_requests_per_minute: 60 http: # Seconds. Applied to the client the bundle creates itself. timeout: 10.0 # Override the indexnowkit-php/ User-Agent. user_agent: null # Service id of a PSR-18 client OR of a symfony/http-client (including # framework.http_client.scoped_clients, wrapped automatically). Default: auto-discovery. # Use a scoped client for proxy, retries or extra headers. client: null key_file: # Serve the key file so engines can verify the key. enabled: true # Route path. {key} is required and constrained to the key format. path: '/{key}.txt' # Restrict the route to this host pattern (a Symfony route host requirement). Default: any host. host: null # Name of the route; rename it when it clashes with an existing one. route_name: indexnowkit_key_file # Cache-Control max-age in seconds. Keep it short so a key rotation propagates quickly. cache_max_age: 300 # Deprecated alias of key_file.enabled. serve_key_file: null # Log the request instead of sending it. Switched on automatically outside prod when no key is set. dry_run: false logging: # Monolog channel every bundle service logs to. channel: indexnow # URLs listed in one log line (the count is always logged). 0 = no URLs in logs (PII policies). max_urls: 20 # Consecutive 403s for one host before the log level escalates to critical (the line to page on). forbidden_escalation: 5 # Bytes of an engine response body kept in a failure log line. max_body: 300 # Override the level of an outcome. Events and their defaults: ok (debug), pending (info), # invalid_request (error), unprocessable (warning), rate_limited (warning), server_error (warning), # unexpected (error), transport (warning), no_key (warning), dry_run (info), disabled (info), # debounced (debug), invalid_url (warning). levels: {} # e.g. { debounced: info, rate_limited: error } resolver: # How many "via:" hops a rule may follow (Comment -> Post -> Author). max_via_depth: 3 # How many related objects one "via:" hop may yield; the rest is dropped with a warning. max_via_fanout: 100 collector: # Flush as soon as this many URLs were collected in one request or command (0 = only at the # end). Bounds memory in long imports; the flush goes through the normal dispatcher. max_urls: 0 # Warn at shutdown about collected URLs that were never flushed. detect_leaks: true profiler: # Register the profiler panel when WebProfilerBundle is present. enabled: true flush: # Listener priority of the kernel.terminate flush. Default -1000: before the profiler (-1024), # so results land in the panel. Raise or lower it to order against your own terminate listeners. priority: -1000 # Same for console.terminate and the Messenger WorkerMessageHandledEvent flush. console_priority: -1024 # Needs indexnowkit/sitemap (composer require indexnowkit/sitemap). Without the package the block is # accepted as is and ignored; indexnow:check says so. sitemap: # Register indexnow:sitemap and the sitemap reader. false = the command does not exist; nothing # else in the bundle reads sitemaps. enabled: true # Sitemap read by indexnow:sitemap when no argument is given. Default: /sitemap.xml. url: null # Levels of followed below the root (0 = the root only). max_depth: 3 # Documents fetched per run, root included. max_sitemaps: 1000 # Size cap of one uncompressed sitemap document (protocol maximum 50 MiB). Documents are # spooled to temp files, never held in memory, so this bounds disk and time rather than RAM. max_bytes: 52428800 # Follow nested sitemaps on other origins (CDN-hosted parts). Off by default: a sitemap then # decides which hosts this server fetches from. --allow-foreign-hosts enables it for one run. allow_foreign_hosts: false # Where a document is kept while parsing. auto = a temp file, or memory when the temp dir is not # writable (read-only container; logged once). disk = a temp file or fail. memory = never touch # the disk (at most max_bytes per document, so memory stays bounded either way). spool: auto # Directory for the temp files. Default: sys_get_temp_dir(), i.e. TMPDIR / sys_temp_dir. On a # readOnlyRootFilesystem point it at an emptyDir or tmpfs mount. spool_dir: null # Extra attempts (1 s, 2 s, 4 s apart) when fetching a document fails on the network or with a # 5xx. 4xx and broken documents are never retried. fetch_retries: 2 doctrine: # Hook Doctrine ORM. Needs indexnowkit/doctrine + doctrine/doctrine-bundle. enabled: true # Lower than Gedmo so slugs exist before URLs are resolved. listener_priority: -100 # Restrict the listener and the commit-safety middleware to these DBAL connection names. # Empty = all connections. connections: [] ``` ## Compile-time validation The container fails to build when: | Rule | Message | |---|---| | `dispatch: messenger` without `base_url` | a Messenger worker has no request context and would generate `http://localhost/...` URLs | | a `hosts` map without `base_url` | the default host would be unknown | | `strict_hosts: true` with neither `base_url` nor `hosts` | there would be no known host at all | | `engines: []` | at least one engine is required | | a literal `key` outside `[A-Za-z0-9-]{8,128}` | | | a literal `base_url` that is not an absolute http(s) URL | | | an `engines` entry that is neither a known engine nor an `http(s)` URL | | | `key_file.path` not starting with `/` or without `{key}` | | | a literal `key_location` that is not an absolute http(s) URL | | | a literal `previous_key` outside `[A-Za-z0-9-]{8,128}` | | | a literal `http.user_agent` containing a line break | | | `logging.levels` with an unknown event | the message lists the known events | | literal `sitemap.url` that is not an absolute http(s) URL | | | `dispatch: messenger` without `symfony/messenger` installed | install it, or use `dispatch: sync` | | a number outside its range: `max_url_length` ≥ 64, `http.timeout` ≥ 0.1, `batch.max_urls` 1–10000, `sitemap.max_bytes` ≥ 1024, `sitemap.max_sitemaps` ≥ 1, `key_file.cache_max_age` ≥ 0, `resolver.max_via_fanout` ≥ 1, `logging.forbidden_escalation` ≥ 1 | the node's own message | Literal values only. A `%env(...)%` placeholder is resolved at runtime, so it is skipped here and validated by the core's `Config` instead — see the next section. Core options with no node here: `retry.*` (Symfony retries through Messenger's `retry_strategy`, so the core `RetryPolicy` is not used by the bundle) and `serve_key_file`, replaced by `key_file.enabled`. ## Environment placeholders and runtime failures Anything read from an environment variable is only known when the container runs. A bad value there — an empty `INDEXNOW_KEY` in production, a malformed `INDEXNOW_BASE_URL` — used to surface as an exception thrown from a Doctrine flush or from `kernel.terminate`. Instead, the bundle logs one `critical` line and runs disabled until the value is fixed: ``` indexnow: invalid configuration, IndexNow is disabled until it is fixed: (run "bin/console indexnow:check") ``` `bin/console indexnow:check` prints the same error and exits non-zero, so it belongs in your deployment smoke tests. ## The `hosts` node and environment variables `hosts` is an array node. Symfony cannot populate a whole array node from a single environment variable, so this does **not** work: ```yaml indexnowkit: hosts: '%env(json:INDEXNOW_HOSTS)%' # not supported ``` List the hosts in YAML and put a placeholder on each key instead: ```yaml indexnowkit: base_url: '%env(INDEXNOW_BASE_URL)%' hosts: example.de: key: '%env(INDEXNOW_KEY_DE)%' base_url: 'https://example.de' ``` ## Per-environment defaults The Flex recipe sets `dry_run: true` in `dev` and `test`. Independently of that, the core turns `dry_run` on by itself whenever no key is configured and `kernel.environment` is not `prod` or `production`, so a developer who never sets `INDEXNOW_KEY` gets logging instead of a boot failure. The reverse is worth an alert: `dry_run` on in production means nothing is being submitted at all. `indexnow:check` reports that combination as an error, not a warning. ## Service aliases Every replaceable piece is a service with an interface alias, so an application can decorate it. | Interface | Service id | |---|---| | `IndexNowKit\IndexNowKit` (facade) | `indexnowkit` | | `Config` | `indexnowkit.config` | | `Http\TransportInterface` | `indexnowkit.transport` (lazy) wrapping `indexnowkit.transport.real` | | `Key\KeyProviderInterface` | `indexnowkit.key_provider` | | `Key\KeyFileResponder` | `indexnowkit.key_file_responder` | | `Url\UrlNormalizerInterface` | `indexnowkit.url_normalizer` | | `Url\UrlResolverInterface` | `indexnowkit.url_resolver` | | `Url\GuardedUrlResolver` | `indexnowkit.guarded_url_resolver` | | `Url\ObjectChangeHandler` | `indexnowkit.change_handler` | | `Url\RouteUrlResolverInterface` | `indexnowkit.route_url_resolver` | | `Url\ResolverLocatorInterface` | `indexnowkit.resolver_locator` | | `Attribute\AttributeReaderInterface` | `indexnowkit.attribute_reader` | | `ClientInterface` | `indexnowkit.client` | | `SubmitterInterface` | `indexnowkit.submitter` | | `Collector\CollectorInterface` | `indexnowkit.collector` | | `Debounce\DebounceStoreInterface` | `indexnowkit.debounce_store` | | `Throttle\ThrottleInterface` | `indexnowkit.throttle` | | `Dispatch\DispatcherInterface` | `indexnowkit.dispatcher` | | `Sitemap\SitemapSourceInterface` (and `Sitemap\SitemapReader`) | `indexnowkit.sitemap_reader` (only with `sitemap.enabled`) | | `Console\SubjectLoaderInterface` (`indexnowkit/console`) | `indexnowkit.entity_loader` (only with Doctrine) | | `Check\CheckerInterface` | `indexnowkit.checker` (runs every `Check\CheckInterface` service, autoconfigured with the `indexnowkit.check` tag) | | `Adapter\SubmitterFactoryInterface` (core) | `indexnowkit.command_submitter_factory` | | `Console\ResultFormatterInterface` (`indexnowkit/console`) | `indexnowkit.result_formatter` | | `Console\Vocabulary`, `Console\*Runner` (`indexnowkit/console`) | `indexnowkit.console.vocabulary`, `indexnowkit.console.{check,submit,submit_entity,explain,sitemap,key_generate}` | | `Check\WiringCheck`, core `Check\SitemapSpoolCheck` | `indexnowkit.check.wiring`, `indexnowkit.check.sitemap_spool` (tagged `indexnowkit.check`) | | `Routing\KeyFileRouteLoader` | `indexnowkit.key_file_routes` | Only `indexnowkit`, `IndexNowKit\IndexNowKit` and the key file controller are public; inject the rest by type where you need them. How to decorate or replace each one: [extending.md](extending.md). ## Container parameters `indexnowkit.dispatch` (the resolved mode, after `auto`), `indexnowkit.messenger.transport`, `indexnowkit.messenger_routed`, `indexnowkit.doctrine_hooked`, `indexnowkit.key_file.path`, `indexnowkit.key_file.host`, `indexnowkit.key_file.route_name`, `indexnowkit.log_channel`. `indexnow:check` prints the first, third and fourth. # Custom resolvers `route:` covers pages that map to a named route. Anything else — a headless front-end, a URL assembled from several sources, one entity that publishes a different page per sales channel — is a resolver. ## Write one ```php namespace App\IndexNow; use IndexNowKit\Event; use IndexNowKit\Url\UrlResolverInterface; final class ProductUrls implements UrlResolverInterface { public function __construct(private readonly ChannelRepository $channels) {} /** @return iterable */ public function resolve(object $subject, Event $event): iterable { \assert($subject instanceof Product); foreach ($this->channels->forProduct($subject) as $channel) { yield $channel->baseUrl . '/p/' . $subject->getSlug(); } } } ``` Return absolute or `base_url`-relative URLs, or nothing at all when the object has no public page. The `$event` argument lets a resolver behave differently for a deletion, which is rarely needed but occasionally is. ## Reference it ```php use IndexNowKit\Attribute\IndexNow; #[IndexNow(resolver: ProductUrls::class)] class Product {} ``` With Symfony's default `services.yaml` (`autoconfigure: true` for everything under `App\`), the class is already a service whose id **is** its FQCN, and the bundle auto-tags every `UrlResolverInterface` service with `indexnowkit.url_resolver`. Nothing else to declare. Reference a service id instead when the resolver is registered under a name: ```yaml # config/services.yaml services: app.product_urls: class: App\IndexNow\ProductUrls arguments: ['@App\Repository\ChannelRepository'] ``` ```php #[IndexNow(resolver: 'app.product_urls')] ``` ## The pitfall: constructor dependencies Lookup happens in this order: 1. the tagged-service locator, by the exact string you wrote; 2. failing that, if the string is a class implementing `UrlResolverInterface`, it is instantiated with `new`. Step 2 only works when the constructor takes no required arguments. A resolver with dependencies that is **not** registered under the id you wrote fails with: > IndexNow resolver App\IndexNow\ProductUrls has constructor dependencies but is not registered as a service under > that id. Register it in services.yaml (autoconfigure: true tags it automatically) or reference the service id in > #[IndexNow(resolver: ...)]. Three ways to hit it, all fixable in one line: - the class lives outside the autoconfigured namespace — register it explicitly; - the service was registered under a different id — reference that id in the attribute; - `autoconfigure: false` in your services file — add the `indexnowkit.url_resolver` tag by hand. A resolver referenced by a string that is neither a tagged service nor an instantiable class fails with *"is neither a UrlResolverInterface service nor an instantiable class"*. Both errors are logged, not thrown: the flush completes and no URLs are produced. Verify the wiring without saving anything: ```bash bin/console debug:container --tag=indexnowkit.url_resolver bin/console indexnow:explain 'App\Entity\Product' 42 ``` ## Resolvers and rule guards A resolver rule is a rule like any other, so it keeps `when`, `fields`, `events`, `via` and `name`: ```php #[IndexNow(resolver: ProductUrls::class, when: 'isListed', fields: ['slug', 'listed', 'channels'])] ``` The guards are applied **before** the resolver runs. A resolver never sees an object whose `when` is false, and never runs for an event the rule does not subscribe to. ## Replacing URL resolution entirely `#[IndexNow(resolver: ...)]` replaces the URLs of one rule. To replace the whole mechanism — no attributes at all, metadata from your own source — decorate the `indexnowkit.url_resolver` service: ```yaml services: App\IndexNow\EverythingResolver: decorates: indexnowkit.url_resolver ``` A hand-written top-level resolver has no rules, so the per-rule guards do not apply to it. `GuardedUrlResolver` keeps the class-level event-subscription check for that case, and still catches everything it throws. If your objects cannot carry attributes but you do want rules, register them at runtime with `IndexNowKit\Attribute\RuleRegistry` and decorate `indexnowkit.attribute_reader` instead. That keeps the whole rule model — guards, per-rule deletions, `via`, locales — and is almost always the better choice. See the [core attribute reference](../core/attribute-reference.md#rules-registered-at-runtime). # Doctrine integration Entity hooks are the reason most people install this bundle: change an entity, commit, and the URLs go out. This page covers the wiring, its limits, and the settings you may need to touch. ## What must be installed ```bash composer require indexnowkit/doctrine doctrine/doctrine-bundle ``` Hooks activate only when **all** of these hold: - `indexnowkit/doctrine` is installed (`IndexNowKit\Doctrine\IndexNowListener` exists); - `doctrine/doctrine-bundle` is registered, or a `doctrine` extension is present in the container; - `indexnowkit.doctrine.enabled` is `true` (the default); - `indexnowkit.enabled` is `true`. Otherwise the bundle still works for manual submission and the commands, and `indexnow:check` says so: ``` ! doctrine: entity hooks are NOT active (needs indexnowkit/doctrine + doctrine/doctrine-bundle, doctrine.enabled: true and enabled: true); use indexnow:submit or $indexNow->submit() ``` `indexnow:submit-entity` and `indexnow:explain` are registered only when Doctrine is available, because they load entities through the manager registry. ## What gets registered | Service | Tag | Purpose | |---|---|---| | `indexnowkit.doctrine.listener` | `doctrine.event_listener` on `onFlush` and `postFlush` | classifies changes per rule, resolves URLs | | `indexnowkit.doctrine.middleware` | `doctrine.middleware` | watches the real COMMIT and ROLLBACK | | `indexnowkit.doctrine.staging` | — | holds URLs until the commit | | `indexnowkit.doctrine.sink` | — | hands committed URLs to the request collector | ## Renamed pages When a field a route parameter reads changes (the slug, the category the path goes through), the old URL now answers 404. The listener resolves the rule against the **previous** values of the change set and announces those URLs as deleted, next to the new URLs as updated, in the same flush (`ObjectChangeHandler::renamed()`, scenario A21). Only route rules; only fields that are writable properties (a `readonly` slug is logged at `debug` and skipped); the old page must have been public (`when` true before the change). ## Listener priority ```yaml indexnowkit: doctrine: listener_priority: -100 ``` The default is `-100`, low enough to run **after** listeners that compute the values URLs depend on. Gedmo Sluggable writes the slug in `onFlush`; if the IndexNow listener ran first it would resolve a route with an empty slug. Lower the value further if you have your own `onFlush` listener that must also run first. Raise it only if you know nothing computes URL-relevant data during the flush. ## Connections and entity managers ```yaml indexnowkit: doctrine: connections: ['default'] # empty = every connection ``` The list scopes both the event listener and the commit-safety middleware to those DBAL connection names. Leave it empty in a single-connection application. Set it when a second connection points at a read replica or a third-party database whose writes have nothing to do with your public pages, so its flushes do not pay for the listener. Names are DBAL connection names as configured under `doctrine.dbal.connections`, and they apply to both tags at once: restricting the listener without the middleware would leave URLs staged and never released. ## Commit safety Doctrine has no after-commit event, and `postFlush` runs **before** the outer `COMMIT` whenever `flush()` is wrapped in `wrapInTransaction()` or a manual transaction. So the listener does not deliver, it stages: 1. `postFlush` resolves the URLs and checks the connection's transaction nesting level. 2. Inside a transaction, the URLs are staged against the **native** connection object. 3. The DBAL driver middleware sees the real `commit()` (nesting level 0, identically in DBAL 3 and 4) and releases them to the collector, or sees `rollBack()` and discards them with a `debug` line. 4. A `commit()` that itself throws also discards, so a pooled connection never delivers them later. 5. Outside a transaction the URLs go to the collector immediately. 6. `kernel.terminate` flushes the collector once, after the response was sent. If a driver exposes no native connection object — an unusual wrapped or custom driver — the listener logs `indexnow: driver has no native connection object; submitting inside an open transaction` and delivers anyway, rather than losing the URLs. ## Version support | | Supported | |---|---| | Doctrine ORM | 2.19+ and 3.x | | DBAL | 3.x and 4.x | | DoctrineBundle | 2.13+ and 3.x | DBAL 4 changed `commit()` and `rollBack()` to return `void` where DBAL 3 returns `bool`; the middleware detects which is present and installs the matching connection wrapper. Nothing to configure. ORM 2 and 3 differ in how a collection reports its mapping (an array versus an object); the listener reads the field name from both. ## What bypasses the unit of work These never reach `onFlush`, so nothing is submitted: - DQL and QueryBuilder bulk `UPDATE` and `DELETE`; - `Connection::executeStatement()` and raw SQL; - `INSERT ... SELECT`; - anything written by another process, a migration, or a database trigger. That is conformance scenario A13 and it is a documented limitation, not a bug: an ORM cannot report changes it never saw. Submit those URLs yourself: ```bash bin/console indexnow:submit-entity 'App\Entity\Post' 42 43 bin/console indexnow:submit-entity 'App\Entity\Post' # every entity, up to --limit bin/console indexnow:submit /posts/hello /posts/world ``` A large migration is better served by regenerating the sitemap and running `indexnow:sitemap --changed-since`. ## Long-running processes A Messenger worker flushes after each handled message, because `FlushListener` also listens to `WorkerMessageHandledEvent`. A console command flushes on `console.terminate`. A custom long-running command — an importer looping over 100 000 rows — gets neither until it exits. The collector grows for the whole run and the batch goes out at the very end, or is lost if the process is killed. Flush periodically: ```php foreach ($rows as $i => $row) { $this->em->persist($this->toEntity($row)); if ($i % 500 === 0) { $this->em->flush(); $this->em->clear(); $this->indexNow->flush(); // hand the collected URLs over } } $this->em->flush(); $this->indexNow->flush(); ``` Also pick the debounce store deliberately for such runs: `cache.app` (the default, shared across processes) keeps the window meaningful, while `memory` is per process and bounded at 50 000 entries. ## Multiple rules on one entity Every `#[IndexNow]` rule of a changed entity is classified separately. During one flush a `Post` can produce an update for its article page and a deletion for its AMP page, because `hasAmp` turned false while `isPublished` stayed true. The listener resolves the deletion in `onFlush`, while the old state is still readable, and the update in `postFlush`. Changed to-many associations are handled too: `post.tags` is not part of the post's change set, so a scheduled collection update re-classifies the owner with the association's field name as the changed field. Use `bin/console indexnow:explain 'App\Entity\Post' 42` to see which rule decided what. # Extending and replacing pieces Every piece of the pipeline is a container service with an interface alias. Three ways to change behaviour, in the order you should try them: 1. **Configuration.** Most knobs are in `indexnowkit.yaml` ([configuration.md](configuration.md)). Prefer it: the value is validated at compile time and shows up in `indexnow:check`. 2. **Decorate.** Keep the shipped implementation, wrap it. Symfony's `#[AsDecorator]` on the service id below is enough; the bundle sees your class through the alias. 3. **Replace.** Register your own implementation of the interface and alias the service id to it. Do this when the shipped implementation is the wrong model, not when you need to tweak it. ## What is replaceable, and how | Piece | Service id | Interface | Config knob | Decorate / replace | |---|---|---|---|---| | HTTP client | `indexnowkit.transport.real` (built by `TransportFactory`) | PSR-18 `ClientInterface` or symfony `HttpClientInterface` | `http.client`, `http.timeout`, `http.user_agent` | point `http.client` at any client service: scoped client, proxy, `RetryableHttpClient`, extra headers | | Transport | `indexnowkit.transport` (lazy, wraps the real one) | `Http\TransportInterface`; `Http\StreamingTransportInterface` for streamed GET | | decorate; see the streaming note below | | Key provider | `indexnowkit.key_provider` | `Key\KeyProviderInterface` | `key`, `hosts`, `key_location`, `previous_key`, `strict_hosts` | replace for keys from a database or a vault; a rotation needs only `previous_key` | | Key file endpoint | `indexnowkit.key_file_responder` + `KeyFileController` | `Key\KeyFileResponder` (final class) | `key_file.*` | disable it and serve the file yourself | | URL normalizer | `indexnowkit.url_normalizer` | `Url\UrlNormalizerInterface` | `base_url`, `max_url_length` | decorate to strip tracking parameters, force a canonical host, or allow only some paths (throw `InvalidUrlException`: the URL becomes a `skipped/invalid_url` result). This is the one hook every path shares, including `--force` / `--dry-run` commands | | URL resolver | `indexnowkit.url_resolver` | `Url\UrlResolverInterface` | `#[IndexNow]` rules, `resolver.max_via_depth`, `resolver.max_via_fanout` | decorate, or add resolvers referenced by `#[IndexNow(resolver:)]` ([custom-resolvers.md](custom-resolvers.md)) | | Attribute reader | `indexnowkit.attribute_reader` | `Attribute\AttributeReaderInterface` | | decorate with `RuleRegistry` to register rules at runtime | | Guarded resolver / change handler | `indexnowkit.guarded_url_resolver`, `indexnowkit.change_handler` | `Url\GuardedUrlResolver`, `Url\ObjectChangeHandler` (final) | | consume them; the ORM listener and the commands do | | Client (HTTP half) | `indexnowkit.client` | `ClientInterface` | `engines`, `hosts..engines`, `batch.max_urls`, `logging.*` | decorate for per-host policy or metrics at request level | | Submitter | `indexnowkit.submitter` | `SubmitterInterface` | `debounce.*`, `dry_run`, `enabled` | decorate (`RetryingSubmitter` is an example); listeners via PSR-14 events. Note: `--force` / `--dry-run` commands build their own `Submitter` through `SubmitterFactory` and bypass a decorator here (they do fire the PSR-14 events) | | Collector | `indexnowkit.collector` | `Collector\CollectorInterface` | `collector.max_urls`, `collector.detect_leaks` | replace for a different per-request store | | Dispatcher | `indexnowkit.dispatcher` | `Dispatch\DispatcherInterface` | `dispatch`, `messenger.transport`, `messenger.delay`, `messenger.stamps` | replace for another queue (`dispatch: none` + your own drain of the collector). A decorator cannot add stamps after the fact: use `messenger.stamps` | | Debounce store | `indexnowkit.debounce_store` | `Debounce\DebounceStoreInterface` | `debounce.store` (any PSR-6 pool, `memory`, `none`), `debounce.key_prefix` | replace | | Throttle | `indexnowkit.throttle` | `Throttle\ThrottleInterface` | `throttle.max_requests_per_minute` | replace for a shared (Redis) limiter | | Sitemap source | `indexnowkit.sitemap_reader` | `Sitemap\SitemapSourceInterface` | `sitemap.*` | decorate to filter or rewrite entries; replace to read from another place or format. Registered only with `indexnowkit/sitemap` installed and `sitemap.enabled: true` (the default) | | Entity loader (commands) | `indexnowkit.entity_loader` | `indexnowkit/console` `Console\SubjectLoaderInterface` | | decorate for soft deletes, tenant scoping, another id format (`byIds()` / `all()` receive the `Event`). Registered only when the Doctrine integration is active ([doctrine.md](doctrine.md)) | | Command submitter (`--force`, `--dry-run`) | `indexnowkit.command_submitter_factory` | core `Adapter\SubmitterFactoryInterface` | | decorate to wrap what the manual commands submit through | | Command output | `indexnowkit.result_formatter` | `indexnowkit/console` `Console\ResultFormatterInterface` | | replace to match your CLI's JSON envelope or table style | | Command bodies | `indexnowkit.console.*` | `indexnowkit/console` `Console\*Runner` | | the commands are input parsing over these; reuse a runner from your own command (a tenant loop over `SubmitSubjectsRunner`) | | `indexnow:check` | `indexnowkit.checker` | `Check\CheckerInterface`; add lines with `Check\CheckInterface` services (autoconfigured) | | add checks rather than replacing the checker | | Key file route | `indexnowkit.key_file_routes` | `Routing\KeyFileRouteLoader` | `key_file.path`, `key_file.host`, `key_file.route_name` | do not import `config/routes.php` and register your own route to `KeyFileController` | | Doctrine listener | `indexnowkit.doctrine.listener` | `IndexNowListener` (final) | `doctrine.*` | disable and write your own on top of `ObjectChangeHandler` (`created()`, `updated()`, `deleted()`, `renamed()`); skip a namespace by decorating the attribute reader to return an empty `RuleSet` | | Flush timing | `indexnowkit.flush_listener` | `EventListener\FlushListener` | `flush.priority`, `flush.console_priority` | order against your own terminate listeners | | Logging | every service, channel `logging.channel` | PSR-3 | `logging.channel`, `logging.levels`, `logging.max_urls`, `logging.forbidden_escalation` | route the channel in `monolog.yaml`; per-outcome levels need no code | Not replaceable on purpose: the spool the sitemap reader parses through (`Sitemap\Spool`), the rule compiler and the result/reason value objects. They are the parts a wrong replacement would silently break, and they have no behaviour worth swapping: configure the spool (`sitemap.spool`, `sitemap.spool_dir`) instead. ## Decorating the sitemap source Keep the shipped reader (fetching, gzip, indexes, retries, spooling) and shape what comes out of it: ```php use IndexNowKit\Sitemap\SitemapSourceInterface; use Symfony\Component\DependencyInjection\Attribute\AsDecorator; use Symfony\Component\DependencyInjection\Attribute\AutowireDecorated; #[AsDecorator('indexnowkit.sitemap_reader')] final class PublicOnlySitemapSource implements SitemapSourceInterface { public function __construct(#[AutowireDecorated] private readonly SitemapSourceInterface $inner) {} public function read(string $sitemap, ?\DateTimeImmutable $changedSince = null): iterable { foreach ($this->inner->read($sitemap, $changedSince) as $entry) { if (!str_starts_with($entry->url, 'https://www.example.com/private/')) { yield $entry; } } } } ``` Yield, do not collect: the command submits every `batch.max_urls` entries while your generator is still running. A replacement that reads from a database or a search index implements the same interface; `$sitemap` is then whatever the command was given (or `sitemap.url`), free for you to interpret. `--allow-foreign-hosts` reaches only the shipped reader, the command warns when it is passed to another source. The shipped reader also accepts a local path or `file://` URL: a sitemap the application writes into `public/` can be read without the web server (`bin/console indexnow:sitemap /var/www/public/sitemap.xml`). Its index parts must then be local files too, or the run needs `--allow-foreign-hosts` to fetch them by URL. ## Decorating the transport ```php use IndexNowKit\Http\Response; use IndexNowKit\Http\StreamingTransportInterface; use Symfony\Component\DependencyInjection\Attribute\AsDecorator; use Symfony\Component\DependencyInjection\Attribute\AutowireDecorated; #[AsDecorator('indexnowkit.transport')] final class MeteredTransport implements StreamingTransportInterface { public function __construct(#[AutowireDecorated] private readonly StreamingTransportInterface $inner, private readonly Meter $meter) {} public function post(string $url, string $json, array $headers = []): Response { $this->meter->count('indexnow.post'); return $this->inner->post($url, $json, $headers); } public function get(string $url): Response { return $this->inner->get($url); } public function download(string $url, $sink): Response { return $this->inner->download($url, $sink); } } ``` Implement `StreamingTransportInterface`, not only `TransportInterface`: a decorator without `download()` still works, but the sitemap reader then falls back to `get()` and buffers each document once (up to 50 MiB) before spooling it. Most needs (proxy, retries, headers, timeouts, mTLS) belong in `http.client` rather than in a transport decorator. ## Replacing the key provider ```php use IndexNowKit\Key\KeyProviderInterface; final class VaultKeyProvider implements KeyProviderInterface { /* keyFor(), keyLocationFor(), isKnownKey(), managedHosts() */ } ``` ```yaml services: App\IndexNow\VaultKeyProvider: ~ indexnowkit.key_provider: '@App\IndexNow\VaultKeyProvider' ``` `indexnow:check`, the key file controller, the profiler panel and `explain` all go through the alias, so they report your keys. ## Submitting from your own domain events Nothing here needs Doctrine. Call the facade from the service that knows the page changed (a CMS publish action, a MongoDB document listener, an import): `$indexNow->submitEntity($page, Event::Updated)` resolves the URLs from the `#[IndexNow]` rules of the object and submits at once; `$indexNow->collect($urls)` + the request-end flush delivers through the configured dispatcher instead. Objects that cannot carry attributes get their rules from `RuleRegistry::registerFor()` (decorate `indexnowkit.attribute_reader`). ## Adding lines to `indexnow:check` A service implementing `Check\CheckInterface` is tagged `indexnowkit.check` by autoconfiguration and runs after the built-in checks; it adds lines, it cannot throw (an exception becomes an error line naming the class): ```php use IndexNowKit\Check\CheckInterface; use IndexNowKit\Check\CheckReport; final class TenantKeysCheck implements CheckInterface { public function __construct(private readonly TenantRepository $tenants) {} public function check(CheckReport $report): void { $missing = $this->tenants->withoutIndexNowKey(); $missing === [] ? $report->ok('tenants: every active tenant has a key') : $report->error(\sprintf('tenants without a key: %s', implode(', ', $missing))); } } ``` ## Conformance tests for your integration `IndexNowKit\Testing\Conformance\CoreConformanceTestCase` (shipped by the core, needs PHPUnit) runs the protocol scenarios of the spec (C01, C03, C04, C06, C09 to C12, C14, C19, C20) against the facade *your* container built. The bundle's own `tests/Functional/CoreConformanceTest.php` is the reference: boot your kernel, return `indexnowkit` and the `FakeTransport` aliased as `indexnowkit.transport`, optionally a second configured host. Run it in the application that decorates or replaces bundle services to prove the pipeline still conforms. ## Listening instead of replacing `Submitter` dispatches every `Result` as a PSR-14 event through `event_dispatcher` when it exists; a listener on `IndexNowKit\Result` gets each outcome for metrics or alerting without touching the pipeline ([operations guide](../core/operations.md)). ## What is the core's `SubmitUrlsHandler` is `Retry\WorkerOutcome` (which URLs to retry, which were rejected for good, the log lines) plus `RecoverableMessageHandlingException` with the engine's Retry-After. The commands configure their arguments and options from `Console\Definitions` and `Sitemap\Console\Definitions` (`CommandDefinition::applyTo($command)`), so `bin/console indexnow:submit-entity --help` matches artisan and Yii2; a custom command over a core runner can call the same `applyTo()`. # HTTP client ## The default With no `http.client` configured, the bundle discovers a PSR-18 client. If `symfony/http-client` is installed it is used and configured with `http.timeout` (10 seconds by default), `max_duration` at twice that, and **no redirects**; Guzzle gets the equivalent treatment. Any other discovered client keeps its own defaults, including its own timeout, because the library will not reconfigure a client it did not create. Redirects are disabled on purpose. An engine endpoint that redirects is a misconfiguration, and following it would hide the real response. Install the pieces if discovery fails: ```bash composer require symfony/http-client nyholm/psr7 ``` The failure is explicit: *"No PSR-18 HTTP client / PSR-17 factories found."* It is raised when the transport is first used, not at boot. ## Built on first use The transport service is a `LazyTransport` wrapping a closure. Discovery and client construction happen at the first outgoing request, so: - a request that submits nothing pays nothing; - a `dry_run` setup never builds a client at all; - `indexnow:check` can report a missing client as a check error instead of crashing. ## Using your own client Point `http.client` at a service id. Two kinds are accepted: ```yaml indexnowkit: http: client: 'app.psr18_client' # a Psr\Http\Client\ClientInterface service ``` ```yaml indexnowkit: http: client: 'my_scoped.client' # a symfony/http-client service, wrapped in Psr18Client automatically ``` Anything else fails at runtime with *"indexnowkit.http.client must be a PSR-18 client or a symfony/http-client service"*. A client you supply is used as it is: `http.timeout` is **not** applied to it, because its own configuration wins. Set the timeout on the client instead. ## Scoped clients The most useful case is a `framework.http_client` scoped client, which gives you proxy settings, retries, extra headers and per-service timeouts without touching the rest of the application. ```yaml # config/packages/framework.yaml framework: http_client: scoped_clients: indexnow.client: scope: 'https://(api\.indexnow\.org|yandex\.com|www\.bing\.com)' timeout: 10 max_duration: 20 max_redirects: 0 headers: Accept: 'application/json' proxy: '%env(default::HTTP_PROXY)%' retry_failed: max_retries: 2 delay: 1000 ``` ```yaml # config/packages/indexnowkit.yaml indexnowkit: http: client: 'indexnow.client' ``` Keep `max_redirects: 0`. And be deliberate about `retry_failed`: it retries transport-level failures underneath the library, which then never sees them. That is fine for connection resets, but leave 429 and 5xx to the library so `Retry-After` and the `Result` taxonomy still apply — Symfony's default retry status codes include 429 and 5xx, so narrow `http_codes` if you enable it. ## Custom endpoints `engines` accepts full endpoint URLs next to the known names, which is how you point at a staging or mock server: ```yaml indexnowkit: engines: ['https://api.indexnow.org/indexnow'] ``` Custom endpoints must use `https`, because the key travels in the request body. Plain `http` is accepted only for loopback hosts (`localhost`, `127.0.0.1`, `[::1]`), which is what test suites use. ## User-Agent The default is `indexnowkit-php/ (+https://github.com/indexnowkit/php)`. Override it with `http.user_agent` when an engine or your own proxy needs to identify the traffic. Line breaks are rejected. ## Timeouts and what they mean `http.timeout` is a per-request timeout applied to the client the bundle creates. A timeout surfaces as a `failed` result with reason `transport`, `retryable: true`, and a warning on the `indexnow` channel. It is never an exception in your application. Ten seconds is generous for an endpoint whose entire job is to accept a JSON list. Lower it if your worker throughput matters more than the occasional slow acceptance; raise it only if you see `transport` failures that disappear on retry. # Async delivery with Messenger Synchronous dispatch sends after the response has been sent, which is already off the critical path. Messenger buys you one thing more: **retries**. A 429 or a 5xx from an engine is retried with back-off by the transport instead of being logged and forgotten. ## The one-line setup Name a transport and the bundle does the routing: ```yaml # config/packages/indexnowkit.yaml indexnowkit: key: '%env(INDEXNOW_KEY)%' base_url: '%env(INDEXNOW_BASE_URL)%' messenger: transport: async ``` That adds `framework.messenger.routing` for `IndexNowKit\SymfonyBundle\Messenger\SubmitUrlsMessage` pointing at the `async` transport, so `messenger.yaml` needs no edit. The transport itself must exist: ```yaml # config/packages/messenger.yaml framework: messenger: failure_transport: failed transports: async: dsn: '%env(MESSENGER_TRANSPORT_DSN)%' retry_strategy: max_retries: 3 delay: 5000 multiplier: 3 failed: 'doctrine://default?queue_name=failed' ``` Then run a worker: `bin/console messenger:consume async`. If you prefer to route it yourself, leave `indexnowkit.messenger.transport` unset and add the entry by hand: ```yaml framework: messenger: routing: 'IndexNowKit\SymfonyBundle\Messenger\SubmitUrlsMessage': async ``` ## `dispatch: auto` `auto` resolves to `messenger` when `symfony/messenger` is installed **and** `framework.messenger.transports` is non-empty; otherwise to `sync`. `enabled: false` forces `none`. That detection is about transports, not about routing. A message that is dispatched but not routed to any transport is handled **synchronously** by the bus, inside `kernel.terminate` — it works, but nothing is retried. That is why `indexnow:check` prints a warning when the mode is `messenger` and `SubmitUrlsMessage` is not routed: ``` ! dispatch is "messenger" but SubmitUrlsMessage is not routed to a transport: it is handled synchronously, 429/5xx are not retried. Set indexnowkit.messenger.transport or add framework.messenger.routing. ``` Set `dispatch: messenger` explicitly in production so the requirement is stated rather than inferred. ## `base_url` is mandatory A worker has no request context. Without `base_url` the router would generate `http://localhost/...` and every relative URL handed to `submit()` would be dropped as invalid. The container refuses to compile with `dispatch: messenger` and no `base_url`. The same applies to per-host `base_url` in multi-domain setups; see [multi-domain.md](multi-domain.md). ## The bus and transactions The default bus is `messenger.default_bus`; change it with `indexnowkit.messenger.bus`. Whatever bus you name must carry the `dispatch_after_current_bus` middleware, which the default buses do. The message is dispatched with `DispatchAfterCurrentBusStamp`, so when the collector is flushed while another message is being handled, the submission is only dispatched after that handler finished. Combined with the `doctrine_transaction` middleware this keeps the guarantee that nothing is announced before its transaction commits. A failure to dispatch never breaks the request: it is logged as `indexnow: cannot dispatch {count} URL(s) to messenger (message {id}), they are lost: {error}` on the `indexnow` channel. ## Retry semantics `SubmitUrlsHandler` submits the batch and then decides: - Nothing retryable — the handler returns. Failures that are final (400, 403, 422) are logged once more at `error` as `indexnow: {count} URL(s) of job {id} rejected permanently ({reasons}); run "bin/console indexnow:check"`; the message is acknowledged and not retried, because retrying a bad key changes nothing. - Something retryable (429, 5xx, network) — it logs `indexnow: {count} URL(s) of job {id} will be retried` at `info` and throws `RecoverableMessageHandlingException`, which hands the decision to the transport's `retry_strategy`. On Symfony 7.2 and later, when the engine sent a `Retry-After`, the largest value in the batch is passed to that exception as an explicit retry delay in milliseconds, so the transport waits as long as the engine asked instead of applying its own back-off. On older versions the transport's `retry_strategy` decides alone. Note what is retried: the **whole batch**, not just the failed URLs. Debouncing keeps that cheap — URLs that succeeded on the first attempt are inside their debounce window and come back as `skipped` rather than being sent again — as long as `debounce.per_url` is larger than your retry delays, which the 600-second default comfortably is. ## When retries are exhausted The bundle adds no dead-letter machinery of its own; it relies on Messenger's. Configure `framework.messenger.failure_transport` and use the standard commands: ```bash bin/console messenger:failed:show bin/console messenger:failed:show -vv # the IndexNow exception message and the URL list bin/console messenger:failed:retry bin/console messenger:failed:remove ``` Without a failure transport an exhausted message is discarded and the URLs are lost with only the worker's log line to show for it. Set one. For a manual resubmission the URLs are in the failed message; `bin/console indexnow:submit ...` sends them directly, and `--force` bypasses the debounce window if the failure is recent. ## The worker flushes too `FlushListener` listens to `WorkerMessageHandledEvent`, so anything a message handler collects — an entity written by a queue job, for example — is submitted after that message is handled, not at the end of the worker process. Long-running custom commands that are not Messenger workers get no such event. Call `$indexNow->flush()` yourself, periodically, or the collector grows for the life of the process and the URLs go out in one batch at the end. ## Monitoring | Signal | Where | |---|---| | `indexnow: {count} URL(s) of job {id} will be retried` | `indexnow` channel, `info`, in the worker | | `indexnow: {count} URL(s) of job {id} rejected permanently ({reasons}); run "bin/console indexnow:check"` | `indexnow` channel, `error`, in the worker | | `indexnow: cannot dispatch {count} URL(s) to messenger (message {id}), they are lost` | `indexnow` channel, `error`, in the web request | | exhausted retries | Messenger's failure transport, default channel | | the URLs a request handed over | Web Profiler panel, which also says results appear in the worker's log | The profiler panel of the dispatching request cannot show HTTP outcomes: by design the request is over before the worker runs. # Multiple domains For IndexNow every host is a separate site with its own key and its own key file. Sub-domains included: `example.com` and `blog.example.com` are two sites, not one. This page works through one application serving three of them: `example.com`, `example.de` and `blog.example.com`. ## Configuration ```yaml # config/packages/indexnowkit.yaml indexnowkit: key: '%env(INDEXNOW_KEY)%' # the key for the base_url host base_url: 'https://example.com' hosts: example.de: key: '%env(INDEXNOW_KEY_DE)%' base_url: 'https://example.de' blog.example.com: key: '%env(INDEXNOW_KEY_BLOG)%' base_url: 'https://blog.example.com' strict_hosts: true ``` Three things to notice. **`base_url` is still required.** It names the default host, the one the top-level `key` belongs to. A `hosts` map without it fails at compile time. **Each host gets its own `base_url`.** A console command or a Messenger worker has no request to inherit a host from. Without a per-host `base_url`, every URL would be generated on the single global one and the German pages would be announced as `https://example.com/...`, which the engine rejects with 422. **`strict_hosts: true` is the safety net.** Without it the default key is used for *any* host you submit, including one that arrived from user input or from a stale database row. With it, a host that is neither `base_url` nor in the map is skipped with reason `no_key` and a warning, instead of being announced under someone else's key. Add `key_location` under a host when its key file cannot live at `/{key}.txt`: ```yaml shop.example.com: key: '%env(INDEXNOW_KEY_SHOP)%' base_url: 'https://shop.example.com' key_location: 'https://shop.example.com/keys/indexnow.txt' ``` It must be on that same host. A `key_location` pointing elsewhere is rejected at configuration time, because engines answer 422 for it. ## Routes with a host Symfony routes can pin a host, and the router bridge honours it: ```php $routes->add('de_article_show', '/artikel/{slug}') ->controller([ArticleController::class, 'show']) ->host('example.de'); ``` Generating that route produces an `example.de` URL, and the client groups by host and picks the German key. Nothing else is needed for this case. ## Rules that pin a host When one entity class serves several domains, pin the host on the rule instead of on the route: ```php use IndexNowKit\Attribute\IndexNow; use IndexNowKit\Attribute\Param\Accessor; #[IndexNow(route: 'page_show', params: ['slug' => 'slug'], host: 'example.de')] // literal #[IndexNow(route: 'page_show', params: ['slug' => 'slug'], host: new Accessor('tenant.domain'))] // per row class Page {} ``` A literal host is used as given. An `Accessor` (or any `Param\*` value) is read from the object, so a multi-tenant `Page` announces itself on its tenant's domain. The bridge then generates on `hosts..base_url` if configured, and on `https://` otherwise. Two rules that differ only by host need different `name` values, or the second overwrites the first: ```php #[IndexNow(route: 'page_show', params: ['slug' => 'slug'], host: 'example.com', name: 'page_com')] #[IndexNow(route: 'page_show', params: ['slug' => 'slug'], host: 'example.de', name: 'page_de')] ``` ## Key files Every host must serve its own key file, and only its own. The bundle's controller reads the request host and asks the key provider whether that key belongs to it, so `https://example.de/.txt` returns 200 while `https://example.de/.txt` returns 404. One tenant cannot serve another tenant's key file, which would otherwise let them claim ownership signals on a domain they do not control. Nothing extra is needed if all three domains reach the same Symfony application. If they do not — a static site on one of them, a CDN in front of another — serve that host's key file there by hand, with `200`, `text/plain` and no redirect. Restrict the route to a single host pattern with `key_file.host` when only one domain should answer it. ## Verifying ```bash bin/console indexnow:check # every managed host bin/console indexnow:check --host example.de # one host bin/console indexnow:check --host example.de --live ``` For each host the report fetches the key file over HTTP and compares the body against the configured key, so a missing file, a redirect, an HTML error page or a stale key after a rotation all surface as an error naming the host and the URL (with the key masked). `--live` additionally POSTs the site root to every engine, even when `dry_run` is on, and reports what each answered. The report also confirms `strict_hosts` is active, which is the line to look for after adding it. ## What a wrong host looks like | Symptom | Cause | |---|---| | `skipped` / `no_key`, warning `skipping N URL(s) for unmanaged host` | the host is not in `hosts` and is not the `base_url` host, with `strict_hosts` on | | `failed` / `unprocessable` (422) | the URLs do not belong to the host they were submitted under, usually a missing per-host `base_url` in a console command or worker | | `failed` / `invalid_key` (403) | that host's key file is not reachable or does not match | | URLs generated as `http://localhost/...` | no `base_url` at all, outside a request | `bin/console indexnow:explain App\Entity\Page 42` prints the host and the key file for every URL it would submit, which is usually faster than reading logs. ## www and apex `example.com` and `www.example.com` are two hosts to IndexNow: each needs its own key file, and a URL submitted under the other one's key answers 422. Pick the canonical one (the one your pages link to and `` names), put it in `base_url`, redirect the other with `301`, and do not list it in `hosts` — listing both would announce two copies of every page. With `strict_hosts: true` a request that reached the application under the non-canonical name submits nothing instead of announcing duplicates. ## hreflang clusters Localized pages that point at each other with `hreflang` are one cluster to the engines: when one changes, announce the cluster. A rule with `locales: 'all'` does that for the locales of one model; for locales living on other hosts `locale_hosts` sends each locale to its host under that host's key. When translations are separate objects, `via:` walks to them: ```php #[IndexNow(route: 'article_show', params: ['slug' => 'slug'], locales: 'all')] // every locale of this article #[IndexNow(via: 'translations')] // or: the sibling objects' own rules ``` # Testing your integration Two questions are worth testing: *would this change be announced?* and *what exactly would be sent?* The first is answered by `dry_run`, the second by swapping the transport for a fake. ## Option 1: dry run The Flex recipe already sets `dry_run: true` in `test`. The whole pipeline runs — rules, guards, URL generation, normalization, host grouping, key lookup, debouncing — and stops before the POST. Results come back as `skipped` with reason `dry_run`. ```php $results = self::getContainer()->get(IndexNowKit::class)->submitEntity($post); self::assertSame(IndexNowKit\Reason::DryRun, $results[0]->reason); self::assertSame(['https://www.example.com/posts/hello'], $results[0]->urls); ``` Good enough for "the rule fires for this entity". It never touches the network and needs no container changes. ## Option 2: fake the transport Alias the transport service to `IndexNowKit\Testing\FakeTransport` in a compiler pass, and the recorded POSTs become assertable. This is the pattern the bundle's own functional tests use. ```php // tests/TestKernel.php protected function build(ContainerBuilder $container): void { $container->addCompilerPass(new class implements CompilerPassInterface { public function process(ContainerBuilder $container): void { $container->setAlias('indexnowkit.transport', FakeTransport::class)->setPublic(true); $container->getDefinition('indexnowkit')->setPublic(true); } }); } protected function configureContainer(ContainerConfigurator $container): void { $container->services()->set(FakeTransport::class)->public(); $container->extension('indexnowkit', [ 'key' => '0123456789abcdef0123456789abcdef', 'base_url' => 'https://www.example.com', 'dispatch' => 'sync', 'debounce' => ['per_url' => 0], ]); } ``` Aliasing `indexnowkit.transport` replaces the lazy wrapper, so discovery never runs and no HTTP client is needed at all in the test environment. Then: ```php public function testPublishingAnArticleSubmitsItsUrl(): void { $client = self::createClient(); $transport = self::getContainer()->get(FakeTransport::class); $client->request('POST', '/articles', ['title' => 'Hello', 'published' => '1']); self::assertResponseIsSuccessful(); self::assertCount(1, $transport->posts); self::assertSame( ['https://www.example.com/en/articles/hello'], $transport->posts[0]['body']['urlList'], ); self::assertSame('www.example.com', $transport->posts[0]['body']['host']); } ``` Set `dry_run: false` for this variant, or nothing is ever posted. ## Making the test deterministic - **Debounce expiry itself** is tested with `IndexNowKit\Testing\FrozenClock`: pass it as `clock:` to a facade built in the test (or register it as the `Psr\Clock\ClockInterface` service) and move time instead of sleeping; see the core's [testing guide](../core/testing.md). - **`debounce: {per_url: 0}`** or `debounce: {store: memory}`. With the default `cache.app` and a real pool, a second test submitting the same URL is silently debounced and the assertion fails for the wrong reason. - **`dispatch: sync`**. Sending happens on `kernel.terminate`, which the test client fires; with `messenger` you would assert on the transport's message queue instead. - **A separate cache directory per kernel variant**, if you boot kernels with different configurations in one suite. Symfony caches the compiled container by environment name, and two variants sharing it will silently reuse the wrong wiring. ## Asserting on the collector instead When you care that URLs were *collected* but not how they were sent, read the collector before terminate, or swap the dispatcher: ```php // configureContainer() $container->services()->set(RecordingDispatcher::class)->public(); // build(), in the compiler pass $container->setAlias('indexnowkit.dispatcher', RecordingDispatcher::class)->setPublic(true); // the test $dispatcher = self::getContainer()->get(RecordingDispatcher::class); self::assertSame(['https://www.example.com/en/articles/hello'], $dispatcher->urls()); ``` `IndexNowKit\Testing\RecordingDispatcher` records every batch handed over and sends nothing, which isolates "the entity hook fired and the transaction committed" from "the HTTP call worked". ## Simulating engine failures ```php $transport->willRespond( new IndexNowKit\Http\Response(429, '', 30), new IndexNowKit\Http\Response(200), ); $transport->willRespond(FakeTransport::failing('connection refused')); // a TransportException ``` Queued responses are consumed in order; anything beyond the queue gets the default `200`. Assert on `$result->reason`, `$result->retryable` and `$result->retryAfter` rather than on message text. For the key file, `$transport->onGet($url, $response)` answers a specific GET; unregistered ones return 404. ## Asserting on logs `IndexNowKit\Testing\ArrayLogger` is the simplest way to check that a rule was skipped for the reason you expect: ```php self::assertStringContainsString('rule "post_amp" skipped', implode("\n", $logger->messages('debug'))); ``` In a full kernel test, the Monolog test handler on the `indexnow` channel does the same job. ## Multi-domain tests Give each host its own key and `base_url`, turn `strict_hosts` on, and add a host-scoped route. Then assert on `$transport->posts[N]['body']['host']` and `['key']`: the client groups URLs by host, so a wrong per-host `base_url` shows up as URLs landing under the wrong key rather than as an exception. ## What not to test The protocol itself is covered by the core's conformance suite (C01–C22) and the bundle's own functional tests (A01–A14, H01–H06). Your suite should cover **your rules**: which entity changes produce which URLs, and that the guards match your publishing workflow. `bin/console indexnow:explain` is the interactive version of the same check. # Troubleshooting [Русская версия](troubleshooting.ru.md) Start here: ```bash bin/console indexnow:check # configuration, key files, engines, wiring bin/console indexnow:explain 'App\Entity\Post' 42 # one entity, the whole decision path, sends nothing ``` Between them they answer most reports without reading a single log line. ## Nothing is sent at all Work down this list; each item names the evidence that confirms it. | Cause | Evidence | |---|---| | `enabled: false` | `skipped` / `disabled`, and one `info` line per call | | `dry_run: true` | `skipped` / `dry_run`, an `info` line with the full request body | | an invalid configuration disabled IndexNow | one `critical` line at boot, and `indexnow:check` exits non-zero | | the entity has no matching rule | `indexnow:explain` says which rule was skipped and why | | the collector was never flushed | `warning: N collected URL(s) discarded: the unit of work ended without flush()` | | Doctrine hooks are not active | `indexnow:check` prints `doctrine: entity hooks are NOT active` | | Messenger is configured but no worker runs | the profiler panel shows URLs dispatched; nothing in the worker log | **Invalid configuration** is the sneaky one. An environment placeholder cannot be validated at compile time, so an empty `INDEXNOW_KEY` in production does not break the container — it logs ``` indexnow: invalid configuration, IndexNow is disabled until it is fixed: (run "bin/console indexnow:check") ``` at `critical` and runs disabled. Alert on that line, and put `indexnow:check` in your deployment smoke tests. ## 403 — the key file By far the most common failure, and it always means the same thing: the engine could not verify `https:///.txt`. ```bash bin/console indexnow:check --host www.example.com curl -i https://www.example.com/.txt ``` The response must be `200`, `text/plain`, body exactly the key, **no redirect**. Frequent causes: - the bundle routes were never imported (no Flex): add `@IndexNowKitBundle/config/routes.php` to `config/routes.yaml`; - `key_file.enabled: false` (or the deprecated `serve_key_file: false`) without a `key_location`; - a CDN or reverse proxy serving a cached old file after a key rotation — `key_file.cache_max_age` defaults to 300 seconds for exactly this reason; - a catch-all route or a firewall matching `/{key}.txt` first; - `http → https` or `www` redirects on the key file; - the request reaching a different host than the one the key belongs to, which the controller answers with 404 by design. The fifth consecutive 403 for a host is logged once at `critical` with *"submissions for this host are not being indexed"*. That is the line to page on. ## 422 — unprocessable The URLs do not belong to the host they were submitted under, or `keyLocation` is invalid. In practice: - a console command or worker generated URLs on the wrong host because a per-host `base_url` is missing — see [multi-domain.md](multi-domain.md); - a rule pinned a `host:` whose key is not configured; - `key_location` points at another host. The configuration layer rejects that, so this only appears with a custom `KeyProviderInterface`. `indexnow:explain` prints the host and key file for every URL it would send, which settles it immediately. ## 429 — rate limited The engine asked you to slow down. The result is `retryable` with `retryAfter` filled when the engine said so. - With Messenger, the transport retries and honours `Retry-After` on Symfony 7.2+. Nothing to do. - With `dispatch: sync`, nothing is retried: the batch is logged and dropped. Switch to Messenger if you see this regularly. - Lower `throttle.max_requests_per_minute` if your own traffic is causing it. Remember it counts per process. - A migration or import announcing tens of thousands of URLs at once is the usual trigger; batch it from a worker. ## A specific URL is never submitted ```bash bin/console indexnow:explain 'App\Entity\Post' 42 --event=updated ``` The output walks the rules in order and prints, per rule, the event subscription, the `when` result, the `fields` filter and the resolved URLs; then per URL the normalized form, the host, the masked key, the key file URL and the debounce state. Common findings: - **`when: false`** — the entity is not published, so the page does not exist. Correct behaviour. - **`fields` filter** — you changed a field the rule does not care about. Add it to `fields`, or drop `fields` to submit on any change. - **not subscribed** — the rule's `events` does not include this event. - **`debounced`** — sent within the last `debounce.per_url` seconds. `--force` on `indexnow:submit` bypasses it. - **`no key for host`** — the host is not in `hosts` and is not the `base_url` host. - **rule errors** — a missing accessor or a route that cannot be generated is printed in red, and logged at `error`. ## Publishing and unpublishing behave oddly `when` is evaluated before and after the change. `true → false` submits a deletion so engines recrawl the 404; `false → true` a creation. Two things to check: - **The backing field must be in the change set.** `when: 'isPublished'` is matched to the `published` field by convention (`isPublished` → `published` → `is_published`, `getStatus` → `status`). When the accessor name has nothing to do with the column, name the fields explicitly with `whenFields: ['status', 'visibleFrom']`. - **Deleting a draft submits nothing**, by design. The page was never public. ## The profiler panel `enabled`, `dry_run`, `dispatch` and `strict_hosts` at the top tell you the mode. Then: - **Collected** — URLs the request buffered. Zero here means no rule produced anything: the problem is upstream, in the rules or the guards. - **Results** — what was actually sent, per engine and host, with the HTTP code and the `reason`. Populated even for synchronous dispatch, which happens after the response. - With Messenger, the results table is empty by design and the panel says HTTP results appear in the worker log. - **Key files** — the URL each managed host must serve, with the key masked. Open one in a browser as a first check. - **base_url** — flagged when unset, because relative URLs and console or worker generation then fail. ## Logs Everything is on the `indexnow` Monolog channel and every message starts with `indexnow: `. ```yaml # config/packages/dev/monolog.yaml monolog: handlers: indexnow: type: stream path: '%kernel.logs_dir%/indexnow.log' level: debug channels: ['indexnow'] ``` Set it to `debug` while diagnosing: the reason a rule decided **not** to produce a URL is only logged at that level. Every message text and level is listed in the [operations guide](../core/operations.md). Keys are masked in every log line, including inside response bodies and exception messages, so these files are safe to attach to a bug report. ## Staging submitted its URLs | Symptom | Cause | Fix | |---|---|---| | Bing/Yandex report URLs of `staging.example.com`, or `failed` / `unprocessable` (422) for them in the log | the staging copy runs with the production key and no `dry_run`; its URLs were generated on its own host | outside production set `INDEXNOW_DRY_RUN=1` (or `INDEXNOW_ENABLED=0`); `check` fails on such a copy since core 0.6 | | the staging host serves the production key file | `key_file.enabled` is on everywhere | `key_file.enabled: false` outside production, so no engine can verify the key on that host | | the engines indexed staging pages | the staging host answered `200` for them and served the key | return `410` (or `noindex` + block in `robots.txt`) on staging, and rotate the key if it was exposed | | a preview environment must submit on purpose | — | say `dry_run: false` explicitly in that environment; `check` then warns instead of failing | ## Duplicates with `memory` and several workers | Symptom | Cause | Fix | |---|---|---| | the same URL is submitted by every worker within minutes | `debounce.store: memory` is per process; each web worker and queue worker keeps its own window | `debounce.store` = a shared cache; `check` warns about `memory` | | duplicates right after a cache outage | the store fails open: no deduplication while the cache is down | expected and bounded (one request per URL); watch the `debounce store unavailable` warning rate | | duplicates after a deploy | the shared cache was flushed, or `debounce.key_prefix` changed | harmless once; keep the prefix stable per application | ## Duplicate submissions after a cache outage The debounce store fails open. If Redis or your cache pool is unavailable, deduplication is skipped and a warning is logged (`debounce store unavailable, submitting without de-duplication`). The visible symptom is a burst of repeated submissions, not lost ones, which is the right trade: a duplicate costs one request, a miss leaves stale content indexed. `indexnow:check` reads a test key through the configured pool and prints `debounce: 600s per URL, shared through cache pool "cache.app" (RedisAdapter)`, or an error naming the pool when it is not usable; `debounce.store: memory` is flagged as per-process. ## `indexnow:sitemap` in a read-only container `SitemapReader` keeps each document in a temp file while parsing. With `readOnlyRootFilesystem` and no writable `/tmp`, the default `sitemap.spool: auto` falls back to memory and logs one warning per run; memory use is bounded by `sitemap.max_bytes` per document (50 MiB), so nothing breaks, it just costs RAM. Mount an `emptyDir` on `/tmp` (or set `TMPDIR` / `sitemap.spool_dir` to a writable volume) to get the temp file back, or set `spool: memory` to make the choice explicit. `indexnow:check` prints where documents are spooled and why the temp dir is unusable. A sitemap that ends midway ("ends early (truncated download or broken sitemap)") or a connection lost during the download is retried `sitemap.fetch_retries` times; when it still fails, the URLs read so far are submitted, the command exits with 1, and a re-run is safe (IndexNow is idempotent, the debounce store absorbs the repeats). For a daily cron with `--changed-since`, use a window wider than the interval (`"2 days"`) so a failed run leaves no gap. ## Getting help Include the output of `bin/console indexnow:check`, the relevant `indexnow` log lines, and the `bin/console indexnow:explain` output for one affected entity. Those three cover almost every question a maintainer would ask. # Backward compatibility `indexnowkit/laravel` follows SemVer. **Before 1.0, minor versions may contain breaking changes**; every one is listed under "Changed" in [CHANGELOG.md](https://github.com/indexnowkit/php/blob/main/packages/laravel/CHANGELOG.md) with the migration. After 1.0 the rules below become the promise. The core's tiers ("call", "implement", "may grow") apply to every core class you touch through this package: [core bc.md](../core/bc.md). ## What the package keeps stable | Surface | Promise | |---|---| | **`config/indexnow.php` keys** ([configuration.md](configuration.md)) | Keys and their meaning stay; new keys are only added with a default. A published config file from an older minor keeps working; a rename ships the old key as deprecated for one minor and is listed in the changelog. | | **Environment variables** the config file reads (`INDEXNOW_KEY`, `INDEXNOW_BASE_URL`, `INDEXNOW_DRY_RUN`, …) | Names stay; new ones are only added. | | **Artisan commands and options** (`indexnow:check`, `indexnow:key:generate`, `indexnow:submit`, `indexnow:submit-model`, `indexnow:explain`, `indexnow:sitemap`) | Names, arguments and options come from the core `Console\Definitions`; new options are only added. Output is not a contract except the exit codes and the `--json` shape of the core formatter. | | **Container bindings** listed in [extending.md](extending.md) | Each abstract stays bound to an implementation of the same interface; rebinding and decorating stay possible. | | **Facade** `Facades\IndexNowKit` and **`IndexNowManager`** (`kit()`, `rules()`, `observe()`, `submit()`, `submitModel()`, `submitModels()`, `urlsFor()`, `urlsForAll()`, `explain()`, `collect()`, `flush()`) | Method names and parameter names stay; new parameters are appended with defaults. | | **Trait `Eloquent\IndexNowable`**, **`Eloquent\IndexNowObserver`**, **`Eloquent\RouteBindingFieldsInterface`** | The trait stays a drop-in; the observer's public hooks keep their names; the interface gets no new method without a major. | | **Queue job** `Queue\SubmitUrlsJob` | Its constructor and the serialized shape stay, so jobs queued before an upgrade still run after it. | | **Route** `indexnow.key_file` (or `key_file.route_name`, `{key}.txt`) | Name and shape stay. | | **Check classes** `Check\QueueCheck`, `Check\EloquentCheck`, `Check\CacheStoreProbe` and the `indexnowkit.check` tag | Names stay; adding a tagged `CheckInterface` keeps working. | Not a contract: log message texts (their `context` keys are), the exact wording the commands print (exit codes and levels are), and the service provider's private methods. ## Pinning `composer require indexnowkit/laravel:^0.9` gets every 0.9.x patch. Read the changelog before a minor. # Configuration `config/indexnow.php` (publish it with `php artisan vendor:publish --tag=indexnow-config`; without the file every key below uses its default). Keys shared with every indexnowkit adapter are described in the [core configuration reference](../core/configuration.md); this page lists all of them briefly and the Laravel-only blocks in full. Values come from `env()`, so they are validated at runtime: an invalid value never throws from a save or from `terminating`. It is logged once at `critical` (`indexnow: invalid configuration, IndexNow is disabled until it is fixed: ...`), IndexNow runs disabled, and `php artisan indexnow:check` prints the exact error. Unknown keys (typos) are logged at `warning`. ## Core keys | Key | Default | Meaning | |---|---|---| | `enabled` | `INDEXNOW_ENABLED`, `true` | kill switch; `false` = nothing is sent, changes are logged at debug | | `key` | `INDEXNOW_KEY` | the IndexNow key, `[A-Za-z0-9-]{8,128}` | | `previous_key` | `INDEXNOW_PREVIOUS_KEY` | the key before a rotation: its file is still served, nothing is submitted under it | | `key_location` | `INDEXNOW_KEY_LOCATION` | full URL of the key file when it is not `https:///.txt` | | `base_url` | `INDEXNOW_BASE_URL`, then `APP_URL` | origin for URLs generated outside HTTP requests (artisan, workers); required with `dispatch: queue` | | `hosts` | `[]` | `host => key` or `host => ['key', 'key_location', 'base_url', 'engines', 'previous_key']` | | `strict_hosts` | `INDEXNOW_STRICT_HOSTS`, `false` | skip URLs of hosts outside `base_url` / `hosts` instead of sending them under the default key | | `environment` | `app()->environment()` | feeds the non-production dry-run safety net | | `production_environments` | `['prod', 'production']` | environments where a missing key is an error, not dry-run | | `max_url_length` | `2048` | longer URLs are `invalid_url` | | `engines` | `INDEXNOW_ENGINES`, `api` | `api`, `yandex`, `bing`, `naver`, `seznam`, `yep`, `internetarchive`, `amazon`, an endpoint URL or an alias | | `engine_aliases` | `[]` | `alias => endpoint URL` | | `locale_hosts` | `[]` | `locale => host` for rules with `locales` and no `host` | | `dispatch` | `INDEXNOW_DISPATCH`, `queue` | `queue`, `sync`, `none` (see below) | | `dry_run` | `INDEXNOW_DRY_RUN`, `false` | log the request instead of sending it | | `batch.max_urls` | `10000` | URLs per request (protocol maximum) | | `debounce.per_url` | `600` | seconds a URL is not resubmitted; `0` = off | | `debounce.key_prefix` | `indexnowkit_` | cache key prefix of the shared window | | `throttle.max_requests_per_minute` | `60` | token bucket per process | | `http.timeout` | `10` | seconds | | `http.user_agent` | `null` | override the `indexnowkit-php/x.y.z` agent | | `retry.*` | `3 / 60 / 2.0 / 3600 / 5` | `max_attempts`, `base_delay`, `multiplier`, `max_delay`, `server_error_delay` of `SubmitUrlsJob` | | `resolver.max_via_depth` / `max_via_fanout` | `3` / `100` | limits of `via:` | | `collector.max_urls` / `detect_leaks` | `0` / `true` | early flush threshold; warn at shutdown about unflushed URLs | | `logging.max_urls` / `forbidden_escalation` / `max_body` / `levels` | `20` / `5` / `300` / `[]` | log line shaping; see the core operations guide | ## Laravel keys ### `dispatch` | Value | Behaviour | |---|---| | `queue` (default) | each flushed batch becomes a `SubmitUrlsJob` on `queue.connection` / `queue.queue`, delayed by `queue.delay` seconds; retries follow `retry.*`. `QUEUE_CONNECTION=sync` runs it inline. | | `sync` | the batch is sent from `app()->terminating()`, after the response; 429/5xx are not retried | | `none` | URLs are collected but never sent (drain `IndexNowKit::kit()->collector` yourself) | ```php 'queue' => [ 'connection' => env('INDEXNOW_QUEUE_CONNECTION'), // null = queue.default 'queue' => env('INDEXNOW_QUEUE'), // null = the connection's default queue 'delay' => 0, // seconds before the first attempt ], ``` Details, Horizon and failure handling: [queue.md](queue.md). ### `debounce.store` | Value | Store | |---|---| | `cache` (default) | `Cache::store()` — the default cache store, shared by web requests and workers | | a store name (`redis`, `array`) | that store | | `memory` | per process; fine for CLI scripts and tests, warned about by `indexnow:check` | | `none` | no debounce | The window is best effort and fails open: an unusable store logs a warning and the URLs are still sent. ### `http.client` `null` discovers a PSR-18 client (Guzzle in a default Laravel application, otherwise php-http/discovery). A class name or container binding of a `Psr\Http\Client\ClientInterface` uses that client (proxy, extra headers, retries of your own). The transport is built on first use, so a request that submits nothing never touches it. ### `key_file` ```php 'key_file' => [ 'enabled' => true, // false = the route is not registered (serve the file yourself or set key_location) 'path' => '/{key}.txt', // must contain {key} 'host' => null, // restrict the route to a host (Route::domain()) 'cache_max_age' => 300, // Cache-Control max-age; short on purpose: a cached old file means 403 after a rotation 'route_name' => 'indexnow.key_file', 'middleware' => [], // no "web" group by default: no session, no CSRF, no cookies ], ``` The controller serves only a key that belongs to the requested host (404 otherwise) and adds `Vary: Host` when a `hosts` map is configured. `serve_key_file` (core name) is accepted as an alias of `key_file.enabled`. The route is compatible with `route:cache`. ### `router` ```php 'router' => [ 'locales' => [], // locales generated for rules with locales: 'all' (empty = current only) 'locale_parameter' => 'locale',// added to the route parameters only when the route declares {locale} 'set_app_locale' => true, // App::setLocale() while generating a locale's URL (localized slugs), restored after ], ``` Inside an HTTP request URLs keep the host Laravel generated them on; in the console (artisan, queue workers) they are rebased onto `base_url`; a rule with `host:` uses `hosts..base_url`, else `https://`. Routes with their own `Route::domain()` keep it. Details: [multi-domain.md](multi-domain.md). ### `eloquent` `eloquent.enabled: false` keeps the observer registered but inert: models using `IndexNowable` submit nothing, manual submission still works. `indexnow:check` says so. ### `logging.channel` Log channel name (`config/logging.php`); `null` = the default channel. Every service of the package logs there, with messages prefixed `indexnow:`. ### `sitemap` Needs `indexnowkit/sitemap` (`composer require indexnowkit/sitemap`); without the package the block is ignored and `indexnow:check` says so. ```php 'sitemap' => [ 'enabled' => true, // false = no indexnow:sitemap command 'url' => null, // default argument (null = /sitemap.xml) 'max_depth' => 3, // nested sitemap indexes 'max_sitemaps' => 1000, 'max_bytes' => 52428800, // per document 'allow_foreign_hosts' => false,// follow index entries on another origin (CDN) 'spool' => 'auto', // auto | disk | memory: where a document is kept while parsing 'spool_dir' => null, // temp directory (null = sys_get_temp_dir()) 'fetch_retries' => 2, // after a network failure or 5xx ], ``` Details: [sitemap.md](sitemap.md). ## Environment variables `INDEXNOW_KEY`, `INDEXNOW_PREVIOUS_KEY`, `INDEXNOW_KEY_LOCATION`, `INDEXNOW_BASE_URL`, `INDEXNOW_ENABLED`, `INDEXNOW_STRICT_HOSTS`, `INDEXNOW_ENGINES` (comma-separated), `INDEXNOW_DISPATCH`, `INDEXNOW_DRY_RUN`, `INDEXNOW_QUEUE_CONNECTION`, `INDEXNOW_QUEUE`, `INDEXNOW_DEBOUNCE_STORE`, `INDEXNOW_LOG_CHANNEL` are read by the shipped config file. Anything else goes through your published copy. ## Startup checks `php artisan indexnow:check` validates the configuration (exact error on failure), fetches every key file over HTTP, and prints the Laravel wiring: the queue connection `SubmitUrlsJob` goes to (or that `sync` retries nothing), whether the debounce cache store is usable, where sitemap documents are spooled, and whether observers are active. `--live` adds a real probe request per engine. # Extending The service provider registers one container binding per core interface. Replace any of them with `$this->app->bind()` / `singleton()` / `extend()` in your own provider (register it after the package, or bind in `register()` — the package resolves everything lazily). ## Bindings | Abstract | Default | Replace it to | |---|---|---| | `IndexNowKit\Config` | `ConfigFactory::create(config('indexnow'))` | — (edit the config) | | `IndexNowKit\Key\KeyProviderInterface` | `StaticKeyProvider::fromConfig()` | keys from a database (multi-tenant); honour `$host` in `isKnownKey()` | | `IndexNowKit\Http\TransportInterface` | `LazyTransport` over a discovered PSR-18 client | your own HTTP stack (or set `http.client`) | | `IndexNowKit\Url\UrlNormalizerInterface` | `UrlNormalizer` | canonical form: strip tracking parameters, trailing-slash policy | | `IndexNowKit\Throttle\ThrottleInterface` | `TokenBucket` | a shared rate limiter | | `IndexNowKit\ClientInterface` | `Client` | — | | `IndexNowKit\Debounce\DebounceStoreInterface` | by `debounce.store` | another window store | | `IndexNowKit\SubmitterInterface` | `Submitter` | wrap with `RetryingSubmitter`, add listeners for metrics | | `IndexNowKit\Collector\CollectorInterface` (scoped) | `Collector` | a durable outbox, a per-tenant buffer | | `IndexNowKit\Attribute\AttributeReaderInterface` / `RuleRegistry` | `RuleRegistry` over `AttributeReader` | your own rule source | | `IndexNowKit\Url\RouteUrlResolverInterface` / `LaravelRouteUrlResolver` | the router bridge | another URL scheme | | `IndexNowKit\Url\ResolverLocatorInterface` | `ContainerResolverLocator` | — | | `IndexNowKit\Url\UrlResolverInterface` | `AttributeUrlResolver` | replace the whole "object → URLs" step | | `IndexNowKit\Url\GuardedUrlResolver`, `ObjectChangeHandler` | the facade's | — | | `IndexNowKit\Dispatch\DispatcherInterface` | by `dispatch` | another delivery (an outbox table, a bus) | | `IndexNowKit\IndexNowKit` | the core facade | — | | `IndexNowKit\Key\KeyFileResponder` | over the key provider | — | | `IndexNowKit\Check\CheckerInterface` | `Checker` with the tagged checks | — | | `IndexNowKit\Sitemap\SitemapSourceInterface` | `SitemapReader` | filter, rewrite or replace the sitemap source (bound only with `indexnowkit/sitemap` installed) | | `IndexNowKit\Laravel\IndexNowManager` | facade root | — | | `IndexNowKit\Laravel\Eloquent\IndexNowObserver` (singleton) | the observer | — | | `IndexNowKit\Laravel\Eloquent\RouteBindingFieldsInterface` | `LaravelRouteUrlResolver` | which model field a `{post:slug}` parameter binds to (drives rename detection) when routes come from elsewhere | | `IndexNowKit\Console\SubjectLoaderInterface` | `Console\ModelLoader` | model lookup (tenant scoping, another id format) | | `IndexNowKit\Console\ResultFormatterInterface` | `ResultRenderer` | command output (your JSON envelope or table style) | | `IndexNowKit\Adapter\SubmitterFactoryInterface` | `SubmitterFactory` | what `--force` / `--dry-run` submit through | | `IndexNowKit\Console\Vocabulary`, `Console\*Runner` | Laravel words; the command bodies | reuse a runner from your own command (a tenant loop over `SubmitSubjectsRunner`) | | `indexnowkit.logger` | `Log::channel(logging.channel)` | a PSR-3 logger of your own (tests: `ArrayLogger`) | ## Custom resolvers ```php #[IndexNow(resolver: ProductUrlResolver::class)] class Product extends Model { use IndexNowable; } final class ProductUrlResolver implements UrlResolverInterface { public function __construct(private readonly UrlGenerator $urls) {} // constructor dependencies are injected public function resolve(object $subject, Event $event): array { return [$this->urls->route('products.show', ['product' => $subject], true)]; } } ``` The class is built by the container; a binding id works the same (`resolver: 'shop.product_urls'`). An unknown id is a logged configuration error, not an exception. ## Rules registered at runtime ```php IndexNowKit::observe(Vendor\Package\Post::class, [new IndexNow(route: 'posts.show', params: ['post' => 'self'])], new IndexNowDefaults(when: 'published')); IndexNowKit::observe(Vendor\Package\Page::class); // the class carries its own attributes, only the observer is missing IndexNowKit::rules()->registerFor(Node::class, fn (Node $node): ?RuleSet => $node->type === 'page' ? RuleCompiler::compile(...) : null); ``` ## Extra checks Anything implementing `IndexNowKit\Check\CheckInterface` and tagged `indexnowkit.check` is printed by `indexnow:check`: ```php $this->app->singleton(CdnPurgeCheck::class); $this->app->tag([CdnPurgeCheck::class], IndexNowKitServiceProvider::CHECK_TAG); ``` Add lines to the report; never throw — a failing check is an error line. ## Submission results Every `Result` (engine, host, status, reason, HTTP code, URL count) is dispatched as a Laravel event, the object itself: `Event::listen(Result::class, fn (Result $result) => ...)` receives it, Telescope's events watcher lists it, `Event::fake()` catches it in tests. The submitter and the command submitters (`--force`, `--dry-run`, the sitemap command) publish through the same PSR-14 bridge (`IndexNowKitServiceProvider::EVENTS`). The lower-level `IndexNowKit::kit()->submitter->addListener(fn (Result $result) => ...)` still works and fires first. `php artisan about` has an `IndexNow` section: core version, enabled/dry-run, environment, base URL, the masked key, engines, dispatch, debounce — the first things a support request needs. `php artisan indexnow:config --json` is the full effective configuration. ## Reading model attributes `#[IndexNow]` accessors on Eloquent models go through `Eloquent\EloquentSubjectReader`, registered with the core's `ParamExtractor::registerReader()`. It claims attributes, casts, accessors and relations (methods with a declared `Relation` return type, or already loaded); anything else falls to the core DSL (methods, properties). An accessor that matches nothing is a `ConfigurationException` logged at `error`, not a silent `null`. ## What is the core's `IndexNowObserver` keeps only what is Eloquent's (the change set from `getChanges()`/`getOriginal()`, the previous state from `getRawOriginal()`, `Connection::afterCommit()`); guarding, logging and the URLs of a row about to be deleted are the core's `Hook\ObserverHelper`. `SubmitUrlsJob` is `Retry\WorkerOutcome` plus `release()`/`fail()` with the delay the `RetryPolicy` computes. The signatures of the artisan commands are rendered from `Console\Definitions` and `Sitemap\Console\Definitions` (`CommandDefinition::laravelSignature()`), so `php artisan indexnow:submit-model --help` matches the bundle and Yii2. A custom command over a core runner can build its signature the same way. # Multiple domains and locales ## One application, several hosts Every host gets its own key (engines verify `https:///.txt` on the submitted host): ```php 'key' => env('INDEXNOW_KEY'), // www.example.com, the base_url host 'base_url' => 'https://www.example.com', 'hosts' => [ 'example.de' => env('INDEXNOW_KEY_DE'), 'shop.example.com' => [ 'key' => env('INDEXNOW_KEY_SHOP'), 'base_url' => 'https://shop.example.com', // origin for this host's URLs outside requests 'engines' => ['yandex', 'bing'], // per-host engine list ], ], 'strict_hosts' => true, // hosts not listed are skipped, not sent under the default key ``` The key file route serves each host's own key only (a request for `example.de`'s key on `www.example.com` is 404) and answers with `Vary: Host`, so a shared CDN never caches one host's file for another. `indexnow:check` fetches every host's key file; `--host=example.de` limits it to one. ## Rules on another host ```php #[IndexNow(route: 'products.show', params: ['product' => 'self'], host: 'shop.example.com')] ``` `host` can also be an accessor (`host: 'tenant.domain'`) for multi-tenant models. The URL is generated through the route and rebased onto `hosts..base_url`, else `https://`. A route with its own `Route::domain()` keeps that domain. ## Locales ```php 'router' => ['locales' => ['en', 'de'], 'locale_parameter' => 'locale', 'set_app_locale' => true], 'locale_hosts' => ['de' => 'example.de'], // optional: one host per locale ``` ```php #[IndexNow(route: 'articles.show', params: ['article' => 'self'], locales: 'all')] ``` - `locales: 'current'` (default) generates one URL; `'all'` one per `router.locales`; a list as given. - The locale is added as the `router.locale_parameter` route parameter **only when the route declares it** (`/{locale}/articles/{article}`), so no query string sneaks in. With `set_app_locale`, `App::setLocale()` is switched for the duration of the generation and restored, which is what packages with localized slugs read. - With `locale_hosts`, a rule without `host` generates each locale on that locale's host and under that host's key. ## Origin of generated URLs | Context | Origin | |---|---| | HTTP request | whatever Laravel generated (the request host) | | artisan, queue worker | `base_url` (Laravel's console request is `APP_URL`; the package rebases scheme, host and port) | | rule with `host:` | `hosts..base_url`, else `https://` | | route with `Route::domain()` | the route's domain, always | A staging copy reached under another hostname would otherwise submit its URLs under the production key; that is what `strict_hosts: true` prevents, and why `indexnow:check` warns when it is off in production. ## www and apex `example.com` and `www.example.com` are two hosts to IndexNow: each needs its own key file, and a URL submitted under the other one's key answers 422. Pick the canonical one (the one your pages link to and `` names), put it in `base_url`, redirect the other with `301`, and do not list it in `hosts` — listing both would announce two copies of every page. With `strict_hosts: true` a request that reached the application under the non-canonical name submits nothing instead of announcing duplicates. ## hreflang clusters Localized pages that point at each other with `hreflang` are one cluster to the engines: when one changes, announce the cluster. A rule with `locales: 'all'` does that for the locales of one model; for locales living on other hosts `locale_hosts` sends each locale to its host under that host's key. When translations are separate objects, `via:` walks to them: ```php #[IndexNow(route: 'article_show', params: ['slug' => 'slug'], locales: 'all')] // every locale of this article #[IndexNow(via: 'translations')] // or: the sibling objects' own rules ``` # Queue, retries, failures `dispatch: queue` is the default. Every batch the collector flushes (end of a request, artisan command or handled job) becomes one `IndexNowKit\Laravel\Queue\SubmitUrlsJob` carrying the normalized URLs. ```php 'dispatch' => 'queue', 'queue' => ['connection' => null, 'queue' => null, 'delay' => 0], 'retry' => ['max_attempts' => 3, 'base_delay' => 60, 'multiplier' => 2.0, 'max_delay' => 3600, 'server_error_delay' => 5], ``` ## What the worker does 1. `SubmitterInterface::submit($urls)`: debounce, group by host, split by `batch.max_urls`, one request per engine. 2. Retryable outcomes (429, 5xx, network failure) → `release($delay)`. The delay is `Retry-After` when the engine sent one, else `base_delay × multiplier^(attempt-1)` for 429 and `server_error_delay × multiplier^(attempt-1)` for 5xx/network, capped at `max_delay`. After `max_attempts` the job **fails** with an error naming the count. 3. Final rejections (400, 403, 422) → the job **fails** immediately with the engine answer in the message and a hint to run `indexnow:check`. 403 means the key file is not reachable under `https:///.txt`. 4. Anything else finishes the job. URLs that were accepted are recorded in the debounce store, so a released job only resends what was rejected. `$job->tries` equals `retry.max_attempts`; `backoff()` mirrors the 5xx schedule for the case the job throws instead of releasing. ## Why failures are failures A 403 is not transient. Recording it in `failed_jobs` is the one place ops already watch; retrying it would hide the broken key file. `php artisan queue:retry ` after fixing the key file re-sends the batch (the URLs are in the payload). ## Horizon, workers, Octane - The job is a plain `ShouldQueue` job: Horizon shows it under its connection and queue, tags are the default ones. - Long-running workers flush the collector after every handled job (`JobProcessed`), so a job that saves models submits their URLs in a follow-up batch on the same connection. - Under Octane the collector is a scoped binding (reset per request) and `terminating` runs per request. ## Synchronous delivery `dispatch: sync` sends from `app()->terminating()`, after the response was written. No retries: a 429 is logged and the URLs are lost until the next change. `QUEUE_CONNECTION=sync` with `dispatch: queue` runs the job inline at flush time; a `release()` on the sync driver retries nothing, so it is a development setting. ## Checking the wiring `php artisan indexnow:check` prints the connection and queue the job goes to, warns when the connection driver is `sync`, and fails when the connection does not exist in `config/queue.php`. # Sitemaps ```bash composer require indexnowkit/sitemap # optional: the indexnow:sitemap command ``` The reader lives in the optional [`indexnowkit/sitemap`](../sitemap/index.md) package. Without it `indexnow:sitemap` exists as a stub that prints `indexnowkit/sitemap is not installed: composer require indexnowkit/sitemap` and exits 1 (a scheduled run gets a sentence, not "command not found"), `indexnow:check` prints `sitemap: not installed (…)`, the `sitemap` block of the config is ignored, and nothing else changes or is logged. `php artisan indexnow:sitemap` submits every URL of a sitemap, a sitemap index, a text sitemap or a gzipped one, reading it as a stream and sending every `batch.max_urls` URLs while it is still being read. A million-URL index never has to fit in memory. ```bash php artisan indexnow:sitemap # sitemap.url, else /sitemap.xml php artisan indexnow:sitemap https://www.example.com/news.xml php artisan indexnow:sitemap storage/app/sitemap.xml # local file, no web server involved php artisan indexnow:sitemap --changed-since="1 day" # only newer than that (entries without lastmod are skipped) php artisan indexnow:sitemap --dry-run # list, send nothing php artisan indexnow:sitemap --json # one row per engine/host/status with url_count and batches ``` Schedule it once a day as a safety net for changes that bypassed the hooks: ```php // routes/console.php Schedule::command('indexnow:sitemap --changed-since="1 day"')->daily(); ``` ## Options | Option | | |---|---| | `--changed-since=` | `"1 day"`, `"2 hours"`, `"2026-09-01"` | | `--allow-foreign-hosts` | follow index entries hosted on another origin (a CDN) for this run; `sitemap.allow_foreign_hosts` makes it permanent | | `-f, --force` | ignore the debounce window | | `--dry-run` | print the entries instead of submitting | | `--json` | machine-readable summary; with `--dry-run` a JSON array of URLs | When the sitemap breaks midway (network, 5xx after `fetch_retries`), the batch read so far is still submitted, the command exits 1 and says how much went out; the re-run is idempotent thanks to the debounce window. ## Configuration ```php 'sitemap' => [ 'enabled' => true, 'url' => null, 'max_depth' => 3, 'max_sitemaps' => 1000, 'max_bytes' => 52428800, 'allow_foreign_hosts' => false, 'spool' => 'auto', 'spool_dir' => null, 'fetch_retries' => 2, ], ``` `spool` decides where a document is kept while parsing: `auto` uses a temp file and falls back to memory when the temp dir is not writable (read-only containers), `disk` fails instead, `memory` never touches the disk (bounded by `max_bytes`). `indexnow:check` reports the location, or why the temp dir is unusable. ## Your own source The command depends on `IndexNowKit\Sitemap\SitemapSourceInterface`. Bind another implementation to read from a database, filter entries or rewrite URLs: ```php $this->app->singleton(SitemapSourceInterface::class, fn ($app) => new FilteringSource($app->make(SitemapReader::class))); ``` `--allow-foreign-hosts` only reaches the shipped `SitemapReader`; a custom source decides on its own. `spatie/laravel-sitemap` users: point `sitemap.url` at the generated file (`public/sitemap.xml`) or pass the path as the argument; the package reads it locally. The reader behind the command is the [`indexnowkit/sitemap`](../sitemap/index.md) package (required by this one); its README documents the `sitemap` block and the safety rules. # Testing your integration The core ships test doubles in `IndexNowKit\Testing` (part of the package, not dev-only): `FakeTransport`, `ArrayLogger`, `FrozenClock`, `RecordingDispatcher`. Bind them into the container before the first submission. ```php use IndexNowKit\Http\TransportInterface; use IndexNowKit\Laravel\IndexNowKitServiceProvider; use IndexNowKit\Testing\{ArrayLogger, FakeTransport}; final class PostPublishingTest extends TestCase { private FakeTransport $transport; protected function setUp(): void { parent::setUp(); $this->app->instance(TransportInterface::class, $this->transport = new FakeTransport()); $this->app->instance(IndexNowKitServiceProvider::LOGGER, new ArrayLogger()); config(['indexnow.dispatch' => 'sync', 'indexnow.debounce.store' => 'memory', 'indexnow.debounce.per_url' => 0]); } public function test_publishing_a_post_notifies_the_engines(): void { $this->post('/posts', ['title' => 'Hello'])->assertCreated(); // the batch leaves on terminate, inside ->post() self::assertSame(['https://www.example.com/posts/hello'], $this->transport->posts[0]['body']['urlList']); self::assertSame('www.example.com', $this->transport->posts[0]['body']['host']); } } ``` Every entry of `$transport->posts` is `['url' => ..., 'json' => ..., 'headers' => ..., 'body' => ...]`, so you assert on `host`, `key`, `keyLocation` and `urlList` directly. `FakeTransport::onGet($url, $response)` stubs key file fetches for `indexnow:check`; `willRespond(new Response(429, '', 30))` queues engine answers. ## Without HTTP at all `IndexNowKit::urlsFor($post)` and `IndexNowKit::explain($post)` resolve the rules of one model without sending; `ObjectChangeHandler::updatedEvents()` (via `IndexNowKit::kit()->changes()`) tells which rule classified an update as created / updated / deleted before any URL exists. ## Queue `Queue::fake()` then `Queue::assertPushed(SubmitUrlsJob::class, fn ($job) => $job->urls === [...])`. To run the worker logic in a test, call `$job->handle($submitter, $logger)` with a mocked `Illuminate\Contracts\Queue\Job` set through `$job->setJob($mock)`; `release()` and `fail()` land on the mock. ## Transactions in tests `RefreshDatabase` / `DatabaseTransactions` wrap each test in a transaction and swap Laravel's transaction manager for the testing one, which runs `afterCommit` callbacks at the test's wrapping level. Submissions therefore behave as in production: they leave when your code's own transaction commits. ## dry_run Outside `production_environments` a missing `INDEXNOW_KEY` switches `dry_run` on: the whole pipeline runs (normalization, deduplication, grouping, key lookup) and stops before the POST, with the body in the `info` log. Set `INDEXNOW_KEY` and `INDEXNOW_DRY_RUN=false` in `phpunit.xml` to exercise the transport in tests. ## Conformance The package's own suite runs the core conformance kits (`IndexNowKit\Testing\Conformance\CoreConformanceTestCase`, `OrmConformanceTestCase`) against the container on `orchestra/testbench`; `tests/Conformance/*` shows the driver an adapter implements. # Troubleshooting [Русская версия](troubleshooting.ru.md) Start with `php artisan indexnow:check`, then `php artisan indexnow:explain "App\Models\Post" `, then the log channel at `debug`. ## Nothing is sent | Symptom | Cause | Fix | |---|---|---| | `check`: `configuration: ...` and exit 1 | an `env()` value is invalid; IndexNow runs disabled | fix the value; the exact error is printed | | `explain`: `when: published -> false` right after `create()` | the `when` attribute only has a database default, so the fresh model does not have it | `protected $attributes = ['published' => false]` on the model, or set it explicitly | | `explain`: `no #[IndexNow] rule` | the model has no attribute and was not registered | add the attribute, or `IndexNowKit::observe()` | | URLs resolved but no POST | `dispatch: queue` and no worker | `php artisan queue:work`, or `dispatch: sync` | | log: `rule "..." ignores this update (fields ...)` | `fields` filter did not match the changed attributes | add the attribute to `fields`, or drop the filter | | log: `Cannot generate route "posts.show": Missing required parameter` | the rule's `params` do not match the route | `params: ['post' => 'self']` for route model binding, or name every parameter | | log: `Cannot read "foo" on App\Models\Post: no method foo(), getFoo(), isFoo() or hasFoo(), no property "foo"…` | typo in an accessor | fix the accessor; attributes, casts, accessors, relations and methods are all valid | | a mass `update()` changed nothing in the index | bulk statements fire no events (A13) | `IndexNowKit::submitModels($query->get())` or `indexnow:submit-model` | | `attach()` on a pivot changed nothing | pivot operations fire no owner events | `$touches = ['posts']` on the related model, rule without `fields` filter | ## Sent, but the engine answers | Answer | Meaning | Fix | |---|---|---| | 403 (`invalid_key`, job failed) | `https:///.txt` is not reachable or has another body | `indexnow:check`; a CDN may cache the old file (`key_file.cache_max_age`) | | 422 (`unprocessable`) | URLs of another host than `host`, or key file on another host | one key per host (`hosts`), `strict_hosts: true` | | 429 (`rate_limited`) | too many requests | the job releases with `Retry-After`; lower `throttle.max_requests_per_minute` | | 202 (`pending`) | accepted, key verification pending | normal for a new key; `check --live` later answers 200 | ## Duplicates, timing - The same URL is not resubmitted within `debounce.per_url` (600 s). `--force` bypasses it; `debounce.store: cache` shares the window between requests and workers, `memory` does not. - Everything from one request leaves as one batch on `terminating`; a job saving models flushes after the job. - A rolled-back transaction submits nothing; a savepoint rollback inside `DB::transaction()` drops only the inner URLs. ## Staging submitted its URLs | Symptom | Cause | Fix | |---|---|---| | Bing/Yandex report URLs of `staging.example.com`, or `failed` / `unprocessable` (422) for them in the log | the staging copy runs with the production key and no `dry_run`; its URLs were generated on its own host | outside production set `INDEXNOW_DRY_RUN=1` (or `INDEXNOW_ENABLED=0`); `check` fails on such a copy since core 0.6 | | the staging host serves the production key file | `key_file.enabled` is on everywhere | `key_file.enabled: false` outside production, so no engine can verify the key on that host | | the engines indexed staging pages | the staging host answered `200` for them and served the key | return `410` (or `noindex` + block in `robots.txt`) on staging, and rotate the key if it was exposed | | a preview environment must submit on purpose | — | say `dry_run: false` explicitly in that environment; `check` then warns instead of failing | ## Duplicates with `memory` and several workers | Symptom | Cause | Fix | |---|---|---| | the same URL is submitted by every worker within minutes | `debounce.store: memory` is per process; each web worker and queue worker keeps its own window | `debounce.store` = a shared cache; `check` warns about `memory` | | duplicates right after a cache outage | the store fails open: no deduplication while the cache is down | expected and bounded (one request per URL); watch the `debounce store unavailable` warning rate | | duplicates after a deploy | the shared cache was flushed, or `debounce.key_prefix` changed | harmless once; keep the prefix stable per application | ## Testing environments Outside `production_environments` a missing key enables `dry_run`: requests are logged, not sent. `check` warns about it; in a production environment `dry_run` is an error. ## Where things are logged Channel `indexnow.logging.channel` (default channel otherwise). Levels: success `debug`, 202 `info`, 403/400 `error`, 422/429/5xx `warning`, invalid configuration `critical`, silent decisions (`when` false, `fields` mismatch) `debug`. The full list is in the [core operations guide](../core/operations.md). # Backward compatibility `indexnowkit/yii2` follows SemVer. **Before 1.0, minor versions may contain breaking changes**; every one is listed under "Changed" in [CHANGELOG.md](https://github.com/indexnowkit/php/blob/main/packages/yii2/CHANGELOG.md) with the migration. After 1.0 the rules below become the promise. The core's tiers ("call", "implement", "may grow") apply to every core class you touch through this package: [core bc.md](../core/bc.md). ## What the package keeps stable | Surface | Promise | |---|---| | **Component options** (`'options' => [...]` of `IndexNowComponent`, [configuration.md](configuration.md)) | Keys and their meaning stay; new keys are only added with a default. A rename ships the old key as deprecated for one minor and is listed in the changelog. | | **Component properties** `options`, `transport`, `debounceStore`, `dispatcher`, `urlResolver`, `logger`, `checks`, `environment`, `sitemapInstalled` and the **accessors** `config()`, `kit()`, `rules()`, `keys()`, `transport()`, `submitter()`, `collector()`, `dispatcher()`, `debounceStore()`, `staging()`, `services()`, `sitemapPackage()`, `sitemapInstalled()` | Names and types stay; new properties and accessors are only added. | | **Console actions and options** (`indexnow/check`, `indexnow/key-generate`, `indexnow/submit`, `indexnow/submit-record`, `indexnow/explain`, `indexnow/sitemap`) | Names, arguments and options come from the core `Console\Definitions`; new options are only added. Output is not a contract except the exit codes and the `--json` shape of the core formatter. | | **Bootstrap** (`'bootstrap' => ['indexnow']`), the controller id `indexnow`, the key file controller `indexnow-key-file` and its URL rule `.txt` | Ids and the rule stay. | | **Behavior `ActiveRecord\IndexNowBehavior`**, **`ActiveRecord\IndexNowObserver`** | The behavior stays a drop-in; the observer's public hooks keep their names. | | **Queue job** `Queue\SubmitUrlsJob` | Its public properties and the serialized shape stay, so jobs pushed before an upgrade still run after it. | | **Check classes** `Check\QueueCheck`, `Check\UrlManagerCheck`, `Check\ActiveRecordCheck`, `Check\CacheProbe` and the `checks` property | Names stay; adding a `CheckInterface` keeps working. | Not a contract: log message texts (their `context` keys are), the exact wording the actions print (exit codes and levels are), and the `Wiring` / `References` helpers the component builds its graph with. ## Pinning `composer require indexnowkit/yii2:^0.7` gets every 0.7.x patch. Read the changelog before a minor. # Commit safety A URL must not reach the engines before the transaction that changed the row commits, or a rolled-back write is announced. Yii2 makes this harder than Laravel or Doctrine: - `yii\db\Connection` fires `EVENT_COMMIT_TRANSACTION` / `EVENT_ROLLBACK_TRANSACTION` **only for the outermost** transaction. Nested `beginTransaction()` calls are savepoints and fire nothing. - The classes involved (`Transaction`, `Schema::createSavepoint()`) are not designed for interception without replacing the connection's command classes in your configuration. ## What the package does 1. In the ActiveRecord event the URLs are resolved while the old state is live (that is why the observer is synchronous, see the core's [adapters guide](../core/adapters.md)). 2. `$db->getTransaction()` is `null`: the change is autocommitted, the URLs go to the collector right away. 3. A transaction is open: the URLs are staged (core `Transaction\VerifyingStaging`) together with a **verifier**, a closure that re-reads the row by primary key with a plain `Query` (bypassing `find()` scopes) and answers whether the change landed: - insert / update / rename: the row exists and carries the written values (loose comparison, driver strings vs typed values); - delete: no row. 4. `EVENT_COMMIT_TRANSACTION`: the verifiers run, the URLs of the changes that landed go to the collector, the rest is dropped with a `debug` line (`discarding N staged URL(s) of Post#7, change not committed`). A change that did not land drops **every** URL it produced, including `via` pages and the old URL of a renamed page: announcing "deleted" for a page that still exists is the one outcome to avoid. 5. `EVENT_ROLLBACK_TRANSACTION`: everything staged on that connection is dropped without a query. Cost: one `SELECT ... WHERE = ?` per changed record, only for changes made inside an explicit transaction. Conformance A02, A05, A05b, A05c pass without touching the connection configuration. ## What it cannot tell - An update inside a rolled-back savepoint whose written values happen to equal the row's current values (`UPDATE ... SET title = title`) passes verification and is sent as a harmless refresh of an existing page. - A verifier that throws (connection gone) counts as landed and is logged at `warning`: a stale URL costs one crawl, a lost one costs the update. - Records without a primary key cannot be verified; their URLs are sent. ## Long-running commands Staged URLs are delivered on the commit event, so an import that commits every 1000 rows sends as it goes. Without transactions the collector fills until the command ends (`collector.max_urls` caps it); call `Yii::$app->indexnow->flush()` between batches if you prefer smaller requests. # Configuration Everything lives in the `options` array of the `indexnow` component. Keys mirror the family-wide schema of `indexnowkit/core` ([configuration.md there](../core/configuration.md)); the blocks marked **Yii** are handled by this package. Verify with `php yii indexnow/check`. ## Core keys Values usually come from `getenv()` in your config file; nothing is read from the environment by the package itself. | Key | Default | Meaning | |---|---|---| | `enabled` | `true` | kill switch; `false` = nothing is sent, changes are logged at debug | | `key` | — | the IndexNow key, `[A-Za-z0-9-]{8,128}` (`php yii indexnow/key-generate`) | | `previous_key` | — | the key before a rotation: its file is still served, nothing is submitted under it | | `key_location` | — | full URL of the key file when it is not `https:///.txt` | | `base_url` | — | origin for URLs generated outside web requests (console, queue workers); required with `dispatch: queue` | | `hosts` | `[]` | `host => key` or `host => ['key', 'key_location', 'base_url', 'engines', 'previous_key']` | | `strict_hosts` | `false` | skip URLs of hosts outside `base_url` / `hosts` instead of sending them under the default key | | `environment` | `YII_ENV` (component property `environment` overrides) | feeds the non-production dry-run safety net and the `check` staging error | | `production_environments` | `['prod', 'production']` | environments where a missing key is an error, not dry-run | | `max_url_length` | `2048` | longer URLs are `invalid_url` | | `engines` | `['api']` | `api`, `yandex`, `bing`, `naver`, `seznam`, `yep`, `internetarchive`, `amazon`, an endpoint URL or an alias | | `engine_aliases` | `[]` | `alias => endpoint URL` | | `locale_hosts` | `[]` | `language => host` for rules with `locales` and no `host` | | `dispatch` | `auto` | `auto`, `queue`, `sync`, `none` (see below) | | `dry_run` | unset | log the request instead of sending it; unset outside production makes `check` fail when a key is configured | | `batch.max_urls` | `10000` | URLs per request (protocol ceiling, not a target) | | `debounce.per_url` | `600` | seconds a URL is not resubmitted; `0` = off | | `debounce.key_prefix` | `indexnowkit_` | cache key prefix of the shared window | | `throttle.max_requests_per_minute` | `60` | token bucket per process | | `http.timeout` | `10` | seconds | | `http.user_agent` | `null` | override the `indexnowkit-php/x.y.z` agent | | `retry.*` | `3 / 60 / 2.0 / 3600 / 5` | `max_attempts`, `base_delay`, `multiplier`, `max_delay`, `server_error_delay` of `SubmitUrlsJob` | | `resolver.max_via_depth` / `max_via_fanout` | `3` / `100` | limits of `via:` | | `collector.max_urls` / `detect_leaks` | `0` / `true` | early flush threshold; warn at shutdown about unflushed URLs | | `logging.max_urls` / `forbidden_escalation` / `max_body` / `levels` | `20` / `5` / `300` / `[]` | log line shaping; see the core operations guide | The full semantics of every core key, and the same table for the other adapters, are in the [core configuration reference](../core/configuration.md) ("One concept, three keys"). ## Yii blocks | Key | Default | Meaning | |---|---|---| | `dispatch` | `auto` | `auto` = `queue` when the queue component exists, else `sync`; or `sync`, `queue`, `none` | | `queue.component` | `queue` | id of the `yiisoft/yii2-queue` component `SubmitUrlsJob` is pushed to | | `queue.ttr` | `300` | time-to-reserve of the job (seconds) | | `queue.delay` | `0` | delay before the first execution (seconds) | | `queue.priority` | `null` | driver priority | | `debounce.store` | `cache` | id of a `yii\caching\CacheInterface` component, `memory` (per process) or `none` | | `http.client` | `null` | component id or class of a PSR-18 client (`null` = discover one) | | `key_file.enabled` | `true` | register the URL rule and controller for `/.txt` | | `key_file.pattern` | `.txt` | the URL rule pattern | | `key_file.cache_max_age` | `300` | `Cache-Control: max-age` of the key file | | `router.languages` | `[]` | languages generated for rules with `locales: 'all'` | | `router.language_parameter` | `language` | GET parameter that carries the language (added whenever a rule has a locale) | | `router.set_app_language` | `true` | switch `Yii::$app->language` while generating a locale's URL, restored afterwards | | `active_record.enabled` | `true` | `false` = `IndexNowBehavior` and the `models` list are inert | | `active_record.models` | `[]` | ActiveRecord classes hooked through class-level events (no behavior needed) | | `sitemap.*` | | needs `indexnowkit/sitemap` (`composer require indexnowkit/sitemap`), else the block is ignored and `indexnow/check` says so: `enabled`, `url`, `max_depth`, `max_sitemaps`, `max_bytes`, `allow_foreign_hosts`, `spool` (auto\|disk\|memory), `spool_dir`, `fetch_retries` | | `logging.category` | `indexnow` | category of the lines written to `Yii::getLogger()`; route it in `log.targets` | ## Component properties Besides `options`, the component accepts replacements as instances, config arrays, class names or component ids (`Instance::ensure`): `transport` (`TransportInterface`), `debounceStore`, `dispatcher`, `urlResolver` (replaces the attribute resolver entirely), `logger` (PSR-3), `checks` (extra `CheckInterface`s for `indexnow/check`), `environment`, `sitemapInstalled` (`null` detects `indexnowkit/sitemap`; `false` runs as if the package were absent — tests, or a deployment that must not read sitemaps). ## Console controller properties `controllerMap['indexnow']` may configure `loader` (`SubjectLoaderInterface`: how `submit-record`/`explain` find records), `formatter` (`ResultFormatterInterface`: table/JSON rendering), `submitters` (`SubmitterFactoryInterface`), `modelNamespaces` (default `['app\models']`). ## Invalid configuration Values come from `getenv()`, so they are only known at runtime. A broken value does not throw from a save or the response: one `critical` line (`indexnow: invalid configuration, IndexNow is disabled until it is fixed: ...`) and IndexNow runs disabled. `php yii indexnow/check` prints the exact error. Unknown keys are logged at `warning`. # Extending ## Replacing pieces Yii2 has no application-wide interface bindings, so the component builds the core graph itself and exposes the seams as properties (an instance, a config array, a class name or a component id): | Property | Interface | Use it for | |---|---|---| | `transport` | `IndexNowKit\Http\TransportInterface` | a recording transport in tests, a proxy | | `debounceStore` | `IndexNowKit\Debounce\DebounceStoreInterface` | a shared store the cache component cannot express | | `dispatcher` | `IndexNowKit\Dispatch\DispatcherInterface` | another queue, an outbox table | | `urlResolver` | `IndexNowKit\Url\UrlResolverInterface` | replace the whole "object → URLs" step | | `logger` | PSR-3 | anything but Yii's logger | | `checks` | list of `IndexNowKit\Check\CheckInterface` | extra lines in `indexnow/check` (a CDN purge, a tenant table) | Underneath, `Wiring` describes the graph once with the core's `Adapter\ServicesBuilder` (the properties above are its overrides, resolved by `References`; the Yii pieces are closures) and `services()` returns the lazy `Adapter\Services`; nothing is built before it is used, and a request that collects nothing builds nothing. Everything built is also readable, as delegates: `kit()`, `config()`, `submitter()`, `collector()`, `keys()`, `transport()`, `debounceStore()`, `routeResolver()`, `observer()`, `staging()`, `checker()`, `rules()`, and `sitemapConfig()` / `sitemapSource()` with the optional `indexnowkit/sitemap` installed (`sitemapInstalled()`; without it they throw a `LogicException` naming the package). ## Custom resolvers ```php #[IndexNow(resolver: ProductUrlResolver::class)] // a class Yii::$container can build #[IndexNow(resolver: 'productUrls')] // or an application component id ``` The class implements `IndexNowKit\Url\UrlResolverInterface`; constructor dependencies come from `Yii::$container`. ## Rules at runtime ```php Yii::$app->indexnow->observe(Product::class, [new IndexNow(route: 'product/view', params: ['id' => 'self'])], new IndexNowDefaults(when: 'active')); Yii::$app->indexnow->rules()->registerFor(Page::class, fn (Page $page): ?RuleSet => ...); // decided per object ``` `observe()` hooks the class through class-level events (`yii\base\Event::on`), the same mechanism as the `active_record.models` list. ## Manual submissions `submit(iterable $urls)`, `submitRecord($record, Event $event)`, `submitRecords(iterable $records)` (one request for many), `urlsFor()`, `explain()` return `Result`s; `collect()` parks URLs in the request collector, `flush()` sends them now. Listen to results the Yii way: `Yii::$app->indexnow->on(IndexNowComponent::EVENT_RESULT, fn (ResultEvent $e) => ...)` (`Event\ResultEvent::$result` is the `Result`; the submitter and the command submitters raise it for every result), or lower-level `Yii::$app->indexnow->submitter()->addListener(fn (Result $r) => ...)`. ## What is the core's The observer keeps only what is Yii's (the change set from `changedAttributes`, the previous state, the verify-on-commit staging); guarding, logging and the URLs of a row about to be deleted are the core's `Hook\ObserverHelper`. `SubmitUrlsJob` is `Retry\WorkerOutcome` plus a delayed re-push of the rejected URLs (the delay of `Retry\RetryPolicy` from `retry.*`, `Retry-After` first; yii2-queue's `canRetry()` only concerns exceptions, see [queue.md](queue.md)). The options of every `indexnow/` come from `Console\Definitions` and `Sitemap\Console\Definitions`, names, shortcuts, descriptions and defaults (`getActionOptionsHelp()` reads them from the definitions, not from the property docblocks), so `php yii help indexnow/submit-record` matches the bundle and artisan. ## Console `controllerMap['indexnow']` accepts `loader` (`IndexNowKit\Console\SubjectLoaderInterface`, tenant scoping or another id format), `formatter` (`ResultFormatterInterface`, your JSON envelope) and `submitters` (`SubmitterFactoryInterface`). The command bodies are the `IndexNowKit\Console\*Runner` of `indexnowkit/console`; a tenant loop over `SubmitSubjectsRunner` is a ten-line controller action. # Multiple domains and languages ## One application, several hosts Every host gets its own key (engines verify `https:///.txt` on the submitted host): ```php 'options' => [ 'key' => getenv('INDEXNOW_KEY'), // www.example.com, the base_url host 'base_url' => 'https://www.example.com', 'hosts' => [ 'example.de' => getenv('INDEXNOW_KEY_DE'), 'shop.example.com' => [ 'key' => getenv('INDEXNOW_KEY_SHOP'), 'base_url' => 'https://shop.example.com', // origin for this host's URLs outside requests 'engines' => ['yandex', 'bing'], // per-host engine list ], ], 'strict_hosts' => true, // hosts not listed are skipped, not sent under the default key ], ``` The key file action serves each host's own key only (a request for `example.de`'s key on `www.example.com` is 404) and answers with `Vary: Host`, so a shared CDN never caches one host's file for another. `php yii indexnow/check` fetches every host's key file; `--host=example.de` limits it to one. ## Rules on another host ```php #[IndexNow(route: 'product/view', params: ['slug' => 'slug'], host: 'shop.example.com')] ``` `host` can also be an accessor (`host: 'tenant.domain'`) for multi-tenant records. The URL is generated through `urlManager` and rebased onto `hosts..base_url`, else `https://`. A URL rule with its own host (`'http://shop.example.com/products/' => 'product/view'`) keeps it. ## Languages ```php 'router' => ['languages' => ['en', 'de'], 'language_parameter' => 'language', 'set_app_language' => true], 'locale_hosts' => ['de' => 'example.de'], // optional: one host per language ``` ```php #[IndexNow(route: 'article/view', params: ['slug' => 'slug'], locales: 'all')] ``` - `locales: 'current'` (default) generates one URL; `'all'` one per `router.languages`; a list as given. - The language is passed as the `router.language_parameter` parameter, so a URL rule that declares it (`'/articles/' => 'article/view'`) puts it in the path; without such a rule it becomes a query parameter — declare the rule. - With `set_app_language`, `Yii::$app->language` is switched for the duration of the generation and restored, which is what localized slugs and `Url::to()` helpers read. - With `locale_hosts`, a rule without `host` generates each language on that language's host and under that host's key. ## Origin of generated URLs | Context | Origin | |---|---| | web request | whatever `urlManager` generated (the request host) | | console command, queue worker | `base_url` (the console `urlManager` has no host; the package rebases scheme, host and port) | | rule with `host:` | `hosts..base_url`, else `https://` | | URL rule with its own host | the rule's host, always | A staging copy reached under another hostname would otherwise submit its URLs under the production key; that is what `strict_hosts: true` prevents, and why `indexnow/check` warns when it is off in production. ## www and apex `example.com` and `www.example.com` are two hosts to IndexNow: each needs its own key file, and a URL submitted under the other one's key answers 422. Pick the canonical one (the one your pages link to and `` names), put it in `base_url`, redirect the other with `301`, and do not list it in `hosts` — listing both would announce two copies of every page. With `strict_hosts: true` a request that reached the application under the non-canonical name submits nothing instead of announcing duplicates. ## hreflang clusters Localized pages that point at each other with `hreflang` are one cluster to the engines: when one changes, announce the cluster. A rule with `locales: 'all'` does that for the locales of one model; for locales living on other hosts `locale_hosts` sends each locale to its host under that host's key. When translations are separate objects, `via:` walks to them: ```php #[IndexNow(route: 'article/view', params: ['slug' => 'slug'], locales: 'all')] // every language of this article #[IndexNow(via: 'translations')] // or: the sibling objects' own rules ``` # Queue, retries, failures `dispatch: auto` (the default) is `queue` when the `queue` component exists. Every batch the collector flushes (end of a request, a console command or a handled job) becomes one `IndexNowKit\Yii2\Queue\SubmitUrlsJob` carrying the normalized URLs, pushed with `queue.ttr`, `queue.delay` and `queue.priority`. ```php 'dispatch' => 'queue', 'queue' => ['component' => 'queue', 'ttr' => 300, 'delay' => 0, 'priority' => null], 'retry' => ['max_attempts' => 3, 'base_delay' => 60, 'multiplier' => 2.0, 'max_delay' => 3600, 'server_error_delay' => 5], ``` ## What the worker does 1. `SubmitterInterface::submit($urls)`: debounce, group by host, split by `batch.max_urls`, one request per engine. 2. Retryable outcomes (429, 5xx, network failure): the job **re-pushes the rejected URLs itself** as a new `SubmitUrlsJob` with the same `id` (the log lines stay connected), `attempt + 1`, the same `ttr`, and a delay, then ends successfully. The delay is `Retry-After` when the engine sent one, else `base_delay × multiplier^(attempt-1)` for 429 and `server_error_delay × multiplier^(attempt-1)` for 5xx/network, capped at `max_delay` (the core's `Retry\RetryPolicy`, from the `retry.*` options of the component). At `retry.max_attempts` the job logs `indexnow: giving up on N URL(s) of job after N attempt(s)` at error and ends. 3. Final rejections (400, 403, 422): `indexnow: N URL(s) of job rejected permanently (...); run "php yii indexnow/check"` at error, the job ends. 403 means the key file is not reachable under `https:///.txt`. 4. Anything else ends the job. URLs that were accepted are recorded in the debounce store, so a re-pushed job only resends what was rejected. The queue's own retry settings (`attempts` of the driver, `ttr`) do not apply to 429/5xx: the job never throws for them. `SubmitUrlsJob::canRetry()` (`RetryableJobInterface`) only concerns exceptions: the core's client never throws on HTTP or network failures, but a custom transport or submitter may throw `Queue\RetryableSubmissionException` to get the driver's retry within `maxAttempts`; any other exception is not retried. ## The sync driver `yii\queue\sync\Queue` stores the pushes and runs them in `run()` (with `handle: true`, at the end of the request), ignoring the delay: the re-pushed attempts run back-to-back in the same `run()`, up to `retry.max_attempts`. Fine for development; `php yii indexnow/check` warns about it. Use db, redis, amqp or another real driver in production. ## Workers Long-running workers flush the collector after every handled job (`Queue::EVENT_AFTER_EXEC` and `EVENT_AFTER_ERROR`), so a job that saves records submits their URLs in a follow-up batch on the same queue. ## Synchronous delivery `dispatch: sync` sends after the response has been sent (or when the command ends). No retries: a 429 is logged and the URLs are lost until the next change. ## Checking the wiring `php yii indexnow/check` prints the dispatch mode and the queue component the job goes to, warns when the component is the sync driver, and fails when `dispatch: queue` names a component that does not exist. # Testing your integration Configure the component with the core's test doubles and read what left: ```php use IndexNowKit\Testing\{ArrayLogger, FakeTransport}; $transport = new FakeTransport(); $logger = new ArrayLogger(); $config['components']['indexnow'] = [ 'class' => \IndexNowKit\Yii2\IndexNowComponent::class, 'options' => ['key' => str_repeat('a', 32), 'base_url' => 'https://www.example.com', 'dispatch' => 'sync', 'debounce' => ['per_url' => 0, 'store' => 'memory']], 'transport' => $transport, 'logger' => $logger, ]; $post = new Post(['slug' => 'hello']); $post->save(false); Yii::$app->indexnow->flush(); // in a request this happens after the response is sent $transport->posts[0]['body']['urlList']; // ['https://www.example.com/posts/hello'] $logger->messages('error'); // [] ``` `FakeTransport::willRespond(new Response(429))` and `onGet($url, new Response(200, $key))` script the engines and the key file; `ArrayLogger::messages($level)` returns interpolated lines. The package's own suite (`tests/`) runs the core conformance kits (`OrmConformanceTestCase`, `CoreConformanceTestCase`) through a web and a console application in memory; `Yii2TestCase` is a template for an application test case: `Fixtures::webApp()` / `consoleApp()` build the application, `Fixtures::destroy()` unregisters Yii's error handler and class-level events between tests. ## Without HTTP at all `Yii::$app->indexnow->kit()->urlsFor($post)` (or `explain($post)`) returns the URLs a record would announce, with the rule that produced each — the assertion for a rule test that should not build a transport. ## Transactions and verify-on-commit Inside `Yii::$app->db->transaction()` (or an explicit `beginTransaction()`), URLs are held with a verifier and re-read by primary key when the transaction commits; a rollback drops everything, an inner rollback the rows it undid. In a test: ```php $tx = Yii::$app->db->beginTransaction(); $post = new Post(['slug' => 'held']); $post->save(false); Yii::$app->indexnow->flush(); self::assertSame([], $transport->posts); // nothing until the commit $tx->commit(); Yii::$app->indexnow->flush(); self::assertCount(1, $transport->posts); // now it left ``` A test that wraps every case in a transaction it rolls back at the end therefore never sees a submission: assert on `urlsFor()` there, or commit. Details: [commit-safety.md](commit-safety.md). ## Queue With `dispatch: queue` (or `auto` with a queue component) the batch becomes a `SubmitUrlsJob`; run it inline with `yii\queue\sync\Queue` (`'queue' => ['class' => \yii\queue\sync\Queue::class, 'handle' => true]`) and assert on the transport after `flush()`, or assert on the pushed job with a driver that records (`handle => false` and read `Yii::$app->queue`). 429/5xx re-pushes are described in [queue.md](queue.md). ## dry_run Outside `production_environments` a missing key enables `dry_run`: the whole pipeline runs — rules, guards, URL generation, host and key selection — and the request is logged (`ArrayLogger::messages('info')` contains the body) instead of sent. Set `dry_run: false` for a test that must reach the transport. ## Conformance `tests/Conformance/` runs the core kits against this package (`CoreConformanceTestCase`: C01–C22 through the component; `OrmConformanceTestCase`: A01–A21 through `IndexNowBehavior`; the H01–H06 assertions `KeyFileAssertions` / `CheckOutputAssertions` through the key file action and the console). An application that replaces a piece (`urlResolver`, `dispatcher`) can extend the same abstract cases to prove nothing regressed. # Troubleshooting [Русская версия](troubleshooting.ru.md) Start with `php yii indexnow/check`, then `php yii indexnow/explain 'app\models\Post' `, then the `indexnow` log category at `debug` (`log.targets[].categories = ['indexnow']`, `levels = ['error', 'warning', 'info', 'trace']`). ## Nothing is sent | Symptom | Cause | Fix | |---|---|---| | `check`: `configuration: ...` and exit 1 | an `options` value (usually from `getenv()`) is invalid; IndexNow runs disabled | fix the value; the exact error is printed and logged once at `critical` | | log: `unknown option(s) in the indexnow configuration: ...` | a typo in `options` (`debounce.per_urls`, `key_file.enabld`) | the dotted path names the key | | `explain`: `when: published -> false` right after `save()` | the `when` attribute only has a database default | set it explicitly before `save()`, or give it a default in `init()` | | `explain`: `no #[IndexNow] rule` | the class has no attribute and was not registered | add `#[IndexNow]`, `IndexNowBehavior`, `active_record.models` or `observe()` | | `explain` yields URLs, the log says nothing on save | the behavior is not attached, or `active_record.enabled` / `enabled` is false | `check` prints `active_record: ... hooked` or the reason | | `debug` log: `did not land on commit` | the change was rolled back with a savepoint, or the verifier could not see the row | expected for a rollback; for a verifier problem see [commit-safety.md](commit-safety.md) | | `debug` log: `debounced` | the URL was sent within `debounce.per_url` | `--force` on a command, or lower the window | | `warning`: `skipping ... unmanaged host` | the URL's host is neither `base_url` nor in `hosts` | add the host to `hosts`, or fix `base_url` | | console: `base_url is not set` | URLs are relative and there is no request | set `base_url`; required with `dispatch: queue` | ## The key file | Symptom | Cause | Fix | |---|---|---| | `GET /.txt` is 404 | pretty URLs are off, or `key_file.enabled` is false, or the key differs | `check` prints the rule; `urlManager.enablePrettyUrl = true` | | the engines answer 403 | the served body is not the submitted key, a redirect, or a cached old file after a rotation | `curl -i https://host/.txt`; `key_file.cache_max_age` is 300 s on purpose | | `check`: `key file ... returned 200` but 403 persists | `hosts` and the submitted host differ (www vs apex) | list every host you submit under `hosts`, set `strict_hosts` | ## Queue | Symptom | Cause | Fix | |---|---|---| | `check`: `queue component "queue" is not configured` | `dispatch: queue` (or `auto` resolved to it) without yii2-queue | install `yiisoft/yii2-queue` and configure the component, or `dispatch: sync` | | 429/5xx attempts run back-to-back, no delay | `yii\queue\sync\Queue` ignores the delay of the re-push | a real driver (db, redis, amqp); `check` warns about `SyncQueue` | | the queue's `attempts` does not limit 429/5xx | the job re-pushes them itself, up to `retry.max_attempts` | set `retry.max_attempts` (and `retry.*` delays) in the component options, see [queue.md](queue.md) | | `ttr` exceeded on large batches | `queue.ttr` (300 s) is too small for the batch count | raise `queue.ttr`, or lower `batch.max_urls` | ## Sitemap `php yii indexnow/sitemap --dry-run` lists what would be sent; `sitemap.enabled is false.` means the block is off or invalid (the log has the reason). `check` prints where documents are spooled; on a read-only filesystem set `sitemap.spool_dir` or `sitemap.spool: memory`. The reader belongs to [`indexnowkit/sitemap`](../sitemap/index.md). ## Sent, but the engine answers | Answer | Meaning | Fix | |---|---|---| | 403 (`invalid_key`, job rejected permanently) | `https:///.txt` is not reachable or has another body | `indexnow/check`; a CDN may cache the old file (`key_file.cache_max_age`) | | 422 (`unprocessable`) | URLs of another host than `host`, or the key file on another host | one key per host (`hosts`), `strict_hosts: true`; console URLs need `base_url` on the right host | | 429 (`rate_limited`) | too many requests | the job re-pushes itself with `Retry-After`; lower `throttle.max_requests_per_minute` | | 202 (`pending`) | accepted, key verification pending | normal for a new key; `check --live` later answers 200 | The 403 counter that escalates to `critical` is per process: several workers each count their own five. ## Duplicates, timing - The same URL is not resubmitted within `debounce.per_url` (600 s). `--force` bypasses it; `debounce.store: cache` shares the window between requests and workers, `memory` does not. - Everything from one request leaves as one batch after the response (`Response::EVENT_AFTER_SEND`); a console command flushes when it ends, a queue worker after every job. - A rolled-back transaction submits nothing; a change re-read on commit that the row does not show is dropped ([commit-safety.md](commit-safety.md)). ## Staging submitted its URLs | Symptom | Cause | Fix | |---|---|---| | Bing/Yandex report URLs of `staging.example.com`, or `failed` / `unprocessable` (422) for them in the log | the staging copy runs with the production key and no `dry_run`; its URLs were generated on its own host | outside production set `INDEXNOW_DRY_RUN=1` (or `INDEXNOW_ENABLED=0`); `check` fails on such a copy since core 0.6 | | the staging host serves the production key file | `key_file.enabled` is on everywhere | `key_file.enabled: false` outside production, so no engine can verify the key on that host | | the engines indexed staging pages | the staging host answered `200` for them and served the key | return `410` (or `noindex` + block in `robots.txt`) on staging, and rotate the key if it was exposed | | a preview environment must submit on purpose | — | say `dry_run: false` explicitly in that environment; `check` then warns instead of failing | ## Duplicates with `memory` and several workers | Symptom | Cause | Fix | |---|---|---| | the same URL is submitted by every worker within minutes | `debounce.store: memory` is per process; each web worker and queue worker keeps its own window | `debounce.store` = a shared cache; `check` warns about `memory` | | duplicates right after a cache outage | the store fails open: no deduplication while the cache is down | expected and bounded (one request per URL); watch the `debounce store unavailable` warning rate | | duplicates after a deploy | the shared cache was flushed, or `debounce.key_prefix` changed | harmless once; keep the prefix stable per application | # Справочник по атрибутам [English version](attribute-reference.md) Класс объявляет **список правил** — по одному на семейство публичных URL, которые у него есть. В PHP это атрибуты; в других языках семейства — декораторы или конфиг-объекты. Все они компилируются в один и тот же `IndexNowKit\Attribute\UrlRule`, и всё ниже по течению — классификация событий, guard'ы, локали, делегирование `via`, дедупликация, вывод `explain` — работает только с ним. ## Три атрибута | Атрибут | Цель | Назначение | |---|---|---| | `#[IndexNow]` | класс, **повторяемый** | одно правило URL | | `#[IndexNowDefaults]` | класс | политика, общая для всех правил класса и его потомков | | `#[IndexNowUrl]` | публичный метод | возвращаемое значение метода — семейство URL (конвенция `get_absolute_url()`) | ```php use IndexNowKit\Attribute\{IndexNow, IndexNowDefaults, IndexNowUrl}; #[IndexNowDefaults(when: 'isPublished', fields: ['slug', 'title', 'body', 'published'])] #[IndexNow(route: 'post_show', params: ['slug' => 'slug'])] #[IndexNow(route: 'post_amp', params: ['slug' => 'slug'], when: 'hasAmp', whenFields: ['ampEnabled'])] class Post {} ``` ## Источники Ровно один источник на `#[IndexNow]`. Ноль источников или два — `ConfigurationException` при компиляции с сообщением, называющим виновников. | Источник | Значение | Даёт | |---|---|---| | `route` | имя маршрута фреймворка | один URL на локаль, сгенерированный `RouteUrlResolverInterface` адаптера | | `resolver` | класс `UrlResolverInterface` или id сервиса | то, что вернёт резолвер | | `via` | аксессор к связанному объекту или коллекции | собственные URL связанных объектов, разрешённые как обновления | | `url` | аксессор, возвращающий `string`, `iterable` или `null` | эти URL | | `urls` | список литеральных URL | эти URL, абсолютные или относительно `base_url` | `url` и `urls` легко перепутать, поэтому проверяются оба: `url: '/about'` и `urls: ['aboutUrl']` отклоняются с сообщением, какой из них вы имели в виду. `resolver` требует `ResolverLocatorInterface`. В чистом PHP это `ArrayResolverLocator`, который также создаёт класс по имени, если у его конструктора нет обязательных аргументов. Адаптеры фреймворков ищут id в контейнере. ## Параметры `params` сопоставляет имя параметра маршрута источнику. Простая строка — DSL аксессоров; всё остальное — один из четырёх типизированных объектов `Param\ParamValue`. ### DSL аксессоров Разрешается в этом порядке, на самом объекте: 1. `'self'` — сам объект (route model binding: `params: ['post' => 'self']`); 2. путь через точку — каждый сегмент разрешается рекурсивно (`'category.slug'`); не-объект в середине бросает исключение; 3. метод с точно таким именем; 4. `get`, `is` или `has` плюс имя с заглавной (`'published'` находит `getPublished()`, затем `isPublished()`, затем `hasPublished()`); 5. свойство, в том числе приватное. Ничего не подошло — `ConfigurationException` с именем аксессора и класса. ### Типизированные источники | Класс | Пример | Смысл | |---|---|---| | `Param\Accessor` | `new Accessor('category.slug')` | явная форма строкового аксессора | | `Param\Value` | `new Value('html')` | константа | | `Param\Formatted` | `new Formatted('publishedAt', 'Y')` | `DateTimeInterface::format()` значения аксессора | | `Param\Call` | `new Call('slugFor', Placeholder::Locale)` | вызов метода; дополнительные аргументы передаются как есть | `Param\Placeholder::Locale` и `Param\Placeholder::Host` подставляются для каждого генерируемого URL, так что `Call` может вернуть slug для локали или путь для тенанта. Извлечение выполняется один раз на URL, а не на правило. ### Приведение типов Параметры маршрута должны годиться для URL. Экстрактор принимает `null` и скаляры как есть, разворачивает `BackedEnum` в его `value`, приводит `Stringable` value object к строке и пропускает обычные объекты для route model binding. Голый `DateTimeInterface` отклоняется с указанием на `new Formatted(...)`: неявное форматирование даты — так URL незаметно меняет форму. Всё остальное бросает исключение. ## Опции правила | Опция | Тип | Дефолт | Смысл | |---|---|---|---| | `when` | имя аксессора, `Condition` (`new Equals(path, value)` или свой класс) или замыкание `fn(object): bool` (только правила времени выполнения) | наследуется | страница существует, пока условие истинно | | `whenFields` | список имён полей | `[]` | поля, стоящие за `when` этого правила, когда имя не совпадает с полем (у `when` уровня класса свои `whenFields` в `#[IndexNowDefaults]`) | | `fields` | список имён полей или `null` | наследуется, затем `[]` | только для обновлений: отправлять, когда изменилось одно из них; `[]` = любое поле | | `events` | подмножество `created`, `updated`, `deleted` (строки или case'ы `Event`) или `null` | наследуется, затем все три | какие события жизненного цикла слушает правило | | `locales` | `'current'`, `'all'` или список, или `null` | наследуется, затем `'current'` | разворачивание локалей для локализованных маршрутов | | `host` | строка или `ParamValue` | `null` | генерировать URL этого правила на этом хосте (мультидомен) | | `name` | строка | выводится | стабильный id правила для логов, `explain` и переопределения в подклассе | `when` — **конъюнкция**: `when` класса и `when` правила должны выполняться оба. `fields`, `events` и `locales` — дефолты, которые правило переопределяет; `null` — наследовать, `[]` — «без фильтра». Строковый аксессор проверяется на truthy, что верно для булевых и неверно для строки статуса (`'draft'` — truthy). Для строковых и enum-состояний используйте `Equals`; он же даёт точное определение старого состояния из change set ORM: ```php use IndexNowKit\Attribute\Param\Equals; #[IndexNow(route: 'post_show', params: ['slug' => 'slug'], when: new Equals('status', 'published'))] #[IndexNow(route: 'job_show', params: ['id' => 'id'], when: new Equals('state', JobState::Open))] // BackedEnum или его value ``` `Equals` — это `Attribute\Param\Condition` (`evaluate(object $subject): bool`); любой класс с этим интерфейсом годится в `when`. Обычный `Condition` не знает старого значения: `ChangeClassifier` вычисляет его на текущем объекте, и переход `open → closed` считается обновлением, а не удалением — если только `whenFields` не называет поле. Реализуйте `FieldCondition` (`field()`, `heldFor($oldValue)`), когда условие читает одно поле: тогда change set даёт точное старое состояние, как у `Equals`. `Equals` в `params` — ошибка типов (условие — не источник значения). `explain` печатает каждое условие со считанным значением (`when: status ("draft") -> true — a non-empty string is truthy; use new Equals('status', "draft")`), `explain --json` — то же одним документом. Подробнее — в английской версии. Правила, зарегистрированные во время выполнения (`RuleRegistry`), могут передать замыкание: `when: fn (WP_Post $p): bool => $p->post_status === 'publish'`. Старое значение замыкания восстановить нельзя, поэтому перечислите читаемые им поля в `whenFields`; изменение одного из них считается сменой видимости (см. таблицу семантики). Неизвестное имя события — `ConfigurationException` с именем атрибута и значением. ### Имена правил Выводятся из источника, если не заданы: имя маршрута; `resolver:`; `via:`; `url:` (и `url:` для `#[IndexNowUrl]`); `urls:<первые два литерала>`. Два правила одного класса с одинаковым выведенным именем получают `#2`, `#3` в порядке объявления. Задавайте явное `name`, когда подкласс должен переопределить конкретное правило или когда выведенное имя нестабильно. ## Дефолты класса и наследование Компилятор обходит иерархию **от корня** к листу. - `#[IndexNowDefaults]` сливается поле за полем, ближайшее объявление побеждает. Объявление со своим `when` заменяет и унаследованные `whenFields`; без `when` — добавляет к ним. - Правила накапливаются. Правило, чьё имя повторяет предка, **заменяет** его; новое имя **добавляет** страницу. Так подкласс меняет одну страницу, не повторяя остальные. - `#[IndexNowUrl]` читается на публичных методах, объявленных каждым классом цепочки, поэтому переопределение в подклассе побеждает. Метод не должен требовать аргументов. - Интерфейсы и трейты **не** сканируются: PHP не наследует атрибуты класса через них, и Doctrine-маппинг ведёт себя так же. ```php #[IndexNowDefaults(when: 'isPublished')] #[IndexNow(route: 'content_show', params: ['slug' => 'slug'])] abstract class Content {} #[IndexNow(route: 'content_show', params: ['slug' => 'slug', 'section' => new Value('news')])] // заменяет #[IndexNow(route: 'news_amp', params: ['slug' => 'slug'])] // добавляет class News extends Content {} ``` В иерархии, где публичные страницы есть лишь у части подклассов, базовый класс не должен нести правил: что не объявлено, то не наследуется. ## Семантика: событие, до, после Видимость оценивается для каждого правила. `W` — конъюнкция `when` класса и `when` правила; `W_before` восстанавливается из change set ORM. | Событие ORM | `W_before` | `W_after` | `fields` совпали | Событие правила | Состояние, из которого строится URL | |---|---|---|---|---|---| | insert | — | true | — | `Created`, если подписано | новое, после записи (id назначены) | | insert | — | false | — | нет | — | | update | true | true | да | `Updated`, если подписано | новое, после записи | | update | true | true | нет | нет | — | | update | true | false | не важно | **`Deleted`**, если подписано | текущее, до записи | | update | false | true | не важно | `Created`, если подписано | новое, после записи | | update | false | false | — | нет | — | | delete | — | true | — | `Deleted`, если подписано | состояние до удаления | | delete | — | false | — | нет | — | | цель `via`, любое событие | — | true | по правилу цели | цель разрешается как `Updated` | собственные правила и guard'ы цели | | цель `via`, любое событие | — | false | — | нет | — | Два следствия стоит проговорить. `fields` никогда не подавляет смену видимости, только обычное обновление: страница, которая только что погасла, объявляется независимо от того, какое поле это сделало. И удаление объекта, чьё правило не применяется, ничего не отправляет — страница никогда не была публичной, чистка черновиков молчит. ### Восстановление `W_before` `ChangeClassifier::classify(UrlRule $rule, object $subject, array $changedFields, array $changeSet = [])` возвращает `Event`, важное правилу, или `null`. Видимость старого состояния — best effort, в три яруса: 1. Аксессор `when`, чьё поле есть в change set, вычисляется **точно** из старого значения. Поле ищется по имени, затем по конвенции: `isPublished → published → is_published`, `hasAmp → amp → has_amp`, `getStatus → status`. `UrlRule::fieldCandidates()` отдаёт этот список. 2. Аксессор без записи в change set, но с зависимым полем (его кандидаты или объявленное `whenFields`) среди изменённых считается **перевернувшимся**. Ложное срабатывание стоит одного запроса; пропуск оставляет мёртвую страницу в индексе. 3. Иначе аксессор сохраняет текущее значение. Называйте поля через `whenFields`, когда аксессор — метод, имя которого не связано с колонкой, например `when: 'isVisibleToPublic', whenFields: ['status', 'visibleFrom']`. ## `via` `via` переотправляет страницы связанного объекта: изменившийся комментарий обновляет свой пост, изменившийся товар — категорию. Цели всегда разрешаются как `Updated`: их страница существует независимо от того, что случилось с источником. Глубина ограничена 3, fan-out — 100 связанных объектов на правило (аргументы конструктора `maxViaDepth` и `maxViaFanout` у `AttributeUrlResolver`); превышение глубины бросает исключение, превышение fan-out пишет warning и останавливается. Правило цели, делегирующее обратно через тот же аксессор, пропускается, так что `A -> B -> A` завершается. Полученные URL сохраняют всю цепочку в имени правила: `via:category -> category_show`. ## Имена полей `fields` и `whenFields` — **имена полей модели**, как их пишет разработчик, никогда не колонки БД. Doctrine `getEntityChangeSet()` даёт ровно их. Объявленное поле совпадает с изменённым, когда они равны или одно — префикс другого через точку, так что `fields: ['address']` ловит изменение embeddable, отражённое как `address.city`. ## Типы, которые потребляют адаптеры ```php final readonly class UrlRule { public string $name; public RuleSource $source; // Route|Resolver|Via|Url|Urls public ?string $route; public array $params; public ?string $resolver; public ?string $via; public ?string $url; public array $urls; public array $when; public array $whenFields; public array $fields; public array $events; public array|string $locales; public string|ParamValue|null $host; public function listensTo(Event $event): bool; public function caresAbout(array $changedFields): bool; public function appliesTo(object $subject): bool; // каждый аксессор `when` truthy public function whenDependsOn(string $field): bool; public static function fieldCandidates(string $accessor): array; } ``` `RuleSet` — все правила одного класса в порядке объявления (сначала родители). Он `Countable` и итерируемый, пуст для классов без правил (вызывающим не нужно проверять null), и даёт `isEmpty()`, `get(string $name)` и `listensTo(Event $event)` как дешёвый предфильтр для ORM-хуков. `AttributeReaderInterface::rules(string|object $classOrObject): RuleSet` — точка поиска. Дефолтный `AttributeReader` компилирует атрибуты через `RuleCompiler` и кэширует по классу на время процесса. На испорченном объявлении он бросает `ConfigurationException` — ORM-хуки должны читать через `ObjectChangeHandler` или `GuardedUrlResolver`, которые логируют. `ResolvedUrl` несёт происхождение для `explain`, логов и панели профайлера: `url`, `rule`, `class`, `event`, `locale`, плюс `source()` (`App\Entity\Post#post_amp`) и `ResolvedUrl::urls()` для сплющивания списка в уникальные строки. ## Правила, регистрируемые во время выполнения Модели, которые не могут нести атрибуты — типы записей CMS, чужие классы, API на замыканиях — используют `RuleRegistry`, реализующий `AttributeReaderInterface` поверх внутреннего ридера (по умолчанию атрибуты). ```php use IndexNowKit\Attribute\{IndexNow, IndexNowDefaults, RuleRegistry}; $registry = new RuleRegistry(); $registry->register(Post::class, [ new IndexNow(route: 'posts.show', params: ['post' => 'self']), new IndexNow(urls: ['/']), ], new IndexNowDefaults(when: 'isPublished')); $registry->register(WP_Post::class, [new IndexNow(resolver: 'wp_permalink')], new IndexNowDefaults( when: fn (WP_Post $post): bool => $post->post_status === 'publish', // или: new Equals('post_status', 'publish') whenFields: ['post_status'], )); $registry->registerFor(CmsPage::class, fn (CmsPage $page): ?RuleSet => $page->isSystem() ? null : $rulesFor($page)); $indexNow = IndexNowKit::create($config, attributes: $registry); ``` `register()` компилирует экземпляры атрибутов, созданные в коде, без рефлексии. `registerFor()` решает по объекту и может вернуть `null`, чтобы передать слово внутреннему ридеру. Зарегистрированные правила заменяют то, что вернул бы внутренний ридер для этого класса, и наследуются подклассами. ## Разрешение без ORM `AttributeUrlResolver` разрешает каждое правило класса через его источник; `GuardedUrlResolver` оборачивает его, чтобы ничего не бросалось. `ObjectChangeHandler` — то, на чём строятся ORM-хуки: классифицирует созданный, изменённый или удалённый объект по правилам и разрешает URL, логируя каждый молчаливый исход. См. [adapters.md](adapters.md). ## Антипаттерны Пять объявлений, которые компилируются, работают и отправляют не то. **1. Литеральный URL в `url:`.** `url:` называет аксессор; `urls:` перечисляет литералы. ```php #[IndexNow(url: '/')] // неверно: читает свойство или метод с именем "/" #[IndexNow(urls: ['/'])] // верно #[IndexNow(url: 'canonicalUrl')] // верно: $post->canonicalUrl() или ->canonicalUrl ``` **2. Строка статуса в `when:`.** Строка — аксессор, читаемый как truthy: `'published'` значит «атрибут `published` truthy», а колонка `status` со значением `'draft'` тоже truthy. ```php #[IndexNow(route: 'post_show', params: ['slug' => 'slug'], when: 'status')] // неверно: 'draft' — truthy #[IndexNow(route: 'post_show', params: ['slug' => 'slug'], when: new Equals('status', 'published'))] // верно ``` **3. Правило на странице, которую нельзя индексировать.** Превью, админка, страница с `noindex` или запрет в `robots.txt`: поисковик её запросит, найдёт неиндексируемой и запишет ошибку в счёт ключа. ```php #[IndexNow(route: 'post_preview', params: ['slug' => 'slug'])] // неверно: превью несёт noindex #[IndexNow(route: 'post_show', params: ['slug' => 'slug'], when: 'isPublished')] // верно: публичная страница под guard'ом ``` **4. Неканонические URL.** Варианты фильтров и сортировок, трекинг-параметры, apex рядом с `www`, `http` рядом с `https`: отправляйте каноническую страницу один раз. Генерируйте URL через роутер с `base_url` на каноническом origin; не склеивайте их строками из запроса. ```php #[IndexNow(urls: ['/products?sort=price&utm_source=indexnow'])] // неверно: вариант и трекинг-параметр #[IndexNow(route: 'products_index')] // верно: каноническая выдача ``` **5. Нет `when` у модели с черновиками.** Без guard'а отправляет каждое сохранение, черновики включительно, а снятая страница объявляется обновлением, не удалением. ```php #[IndexNow(route: 'post_show', params: ['slug' => 'slug'])] // неверно, если у Post есть черновики #[IndexNowDefaults(when: 'isPublished')] // верно: черновики пропускаются, #[IndexNow(route: 'post_show', params: ['slug' => 'slug'])] // published → draft — удаление ``` Что библиотека проверяет за вас: URL абсолютный `http(s)`, без credentials, фрагмента и управляющих символов, и принадлежит хосту, для которого у вас есть ключ (`strict_hosts`). Что проверить не может: `noindex`, `robots.txt`, canonical на другой адрес, код ответа страницы — сегодня это забота автора правила, а в следующих релизах — дополнения `verify` (`check --sample`). # Конфигурация [English version](configuration.md) — там же генерируемая таблица «One concept, three keys» (ключи трёх адаптеров). `IndexNowKit\Config` — неизменяемый value object, общий для всех адаптеров. Строится одним из трёх способов и валидируется в конструкторе, поэтому сломанная настройка падает при загрузке, а не при первой отправке. ```php use IndexNowKit\Config; $config = Config::fromArray([...]); // конфиг-файлы фреймворка $config = Config::fromEnv(); // переменные окружения INDEXNOW_* $config = new Config(key: '...', baseUrl: '...'); // именованные аргументы $config = $config->with(dryRun: true); // неизменяемая копия ``` ## Опции `fromArray()` принимает вложенную форму ниже; это каноническая схема, которую повторяет каждый языковой адаптер. ```php Config::fromArray([ 'enabled' => true, 'key' => $_ENV['INDEXNOW_KEY'], 'hosts' => [ 'www.example.com' => 'KEY-FOR-EXAMPLE', 'shop.example.com' => [ 'key' => 'KEY-FOR-SHOP', 'key_location' => 'https://shop.example.com/keys/indexnow.txt', 'base_url' => 'https://shop.example.com', ], ], 'strict_hosts' => true, 'key_location' => null, 'base_url' => 'https://www.example.com', 'engines' => ['api'], 'dispatch' => 'sync', 'batch' => ['max_urls' => 10000], 'debounce' => ['per_url' => 600], 'throttle' => ['max_requests_per_minute' => 60], 'http' => ['timeout' => 10.0, 'user_agent' => null], 'key_file' => ['enabled' => true], 'dry_run' => false, 'environment' => $_ENV['APP_ENV'] ?? null, ]); ``` | Опция | Аргумент конструктора | Дефолт | Смысл | |---|---|---|---| | `enabled` | `enabled` | `true` | `false` отбрасывает каждую отправку; URL возвращаются как `skipped` с причиной `disabled`, лог на `info` | | `key` | `key` | `null` | ключ по умолчанию, 8–128 символов `[A-Za-z0-9-]`, для каждого хоста, не перечисленного в `hosts` | | `hosts` | `hosts` | `[]` | `host => key` или `host => {key, key_location?, base_url?, engines?, previous_key?}` | | `strict_hosts` | `strictHosts` | `false` | применять ключ по умолчанию **только** к хосту `base_url`; любому другому хосту нужна запись в `hosts`, иначе его URL пропускаются | | `key_location` | `keyLocation` | `null` | абсолютный URL файла ключа, если он не `https://{host}/{key}.txt` | | `base_url` | `baseUrl` | `null` | абсолютный URL сайта; разрешает относительные URL, обязателен вне HTTP-запроса | | `engines` | `engines` | `['api']` | имена движков (`api`, `yandex`, `bing`, `naver`, `seznam`, `yep`, `internetarchive`, `amazon`) или полные URL endpoint'ов | | `dispatch` | `dispatch` | `'sync'` | режим доставки, определяемый адаптером; ядро проверяет идентификатор и печатает его | | `batch.max_urls` | `batchMaxUrls` | `10000` | URL на запрос; `Config::MAX_BATCH_URLS` — максимум протокола, потолок, а не цель: меньшие батчи принимаются так же | | `debounce.per_url` | `debouncePerUrl` | `600` | секунды, в течение которых тот же URL не отправляется повторно; `0` выключает дебаунс | | `throttle.max_requests_per_minute` | `throttleMaxRequestsPerMinute` | `60` | исходящих запросов в минуту на процесс; `0` = без лимита | | `http.timeout` | `httpTimeout` | `10.0` | секунды, только для клиентов, которые библиотека создаёт сама | | `http.user_agent` | `userAgent` | `null` | заменяет `indexnowkit-php/ (+https://github.com/indexnowkit/php)` | | `key_file.enabled` | `serveKeyFile` | `true` | должен ли адаптер отвечать на `GET /{key}.txt`; `serve_key_file` — устаревшее имя, побеждает, если заданы оба | | `key_file.cache_max_age` | `keyFileMaxAge` / `keyFileHeaders()` | `300` | `Cache-Control: max-age` ответа с файлом ключа; короткий нарочно — закэшированный старый файл после ротации превращает каждую отправку в 403 | | `debounce.store` | `debounceStore` | `null` | `memory` (на процесс), `none` или id, который адаптер разрешает в свой общий кэш; `null` = дефолт адаптера (Laravel `cache`, бандл `cache.app`, Yii2 `cache`, чистый PHP `memory`) | | `http.client` | `httpClient` | `null` | id или класс PSR-18 клиента, который разрешает адаптер; `null` = discovery | | `dry_run` | `dryRun` | `false` | логировать запрос вместо отправки; вне production незаданный `dry_run` при настроенном ключе делает `check` красным | | `environment` | `environment` | `null` | окружение приложения; управляет страховкой ниже | | `production_environments` | `productionEnvironments` | `['prod', 'production']` | имена окружений (без учёта регистра), считающихся production; заменяет список по умолчанию | | `previous_key` | `previousKey` | `null` | ключ до ротации: файл ключа его ещё принимает, отправки под ним не идут; также `hosts..previous_key` | | `hosts..engines` | `hostEngines` / `endpointsFor()` | наследует `engines` | движки только для одного хоста | | `engine_aliases` | `engineAliases` / `resolveEngine()` | `{}` | короткие имена для своих endpoint'ов, применимы везде, где называется движок | | `locale_hosts` | `localeHosts` / `hostForLocale()` | `{}` | локаль => хост; правила с `locales` и без `host` генерируют каждую локаль на её хосте | | `logging.max_body` | `logBody` | `300` | байт тела ответа движка, сохраняемых в строке лога об ошибке | | `max_url_length` | `maxUrlLength` | `2048` | URL длиннее пропускаются как `invalid_url` | | `debounce.key_prefix` | `debounceKeyPrefix` | `'indexnowkit_'` | префикс ключей общего debounce store | | `logging.max_urls` | `logUrls` / `logSample()` | `20` | URL, перечисляемых в одной строке лога; `0` = только счётчики | | `logging.forbidden_escalation` | `forbiddenEscalation` | `5` | подряд идущих 403 на хост до эскалации лога в `critical` | | `logging.levels` | `logLevels` / `logLevel()` | `{}` | переопределение PSR-3 уровня по исходу; события и дефолты в `Config::LOG_EVENTS` | | `retry.max_attempts`, `retry.base_delay`, `retry.multiplier`, `retry.max_delay`, `retry.server_error_delay` | `retryPolicy()` | `3`, `60`, `2.0`, `3600`, `5` | `RetryPolicy` для обработчиков очередей и `RetryingSubmitter` | | `resolver.max_via_depth`, `resolver.max_via_fanout` | `resolverMaxViaDepth`, `resolverMaxViaFanout` | `3`, `100` | лимиты обхода `via:` в `AttributeUrlResolver`; `IndexNowKit::create()` этот резолвер не строит — адаптер, который строит, передаёт их вместе с `localeHosts` | | `collector.max_urls` | `collectorMaxUrls` | `0` | `IndexNowKit::collect()` сбрасывает буфер досрочно при этом размере; `0` = только на `flush()` | | `collector.detect_leaks` | `collectorDetectLeaks` | `true` | warning при завершении о собранных, но не сброшенных URL | Константы, на которые стоит ссылаться вместо чисел: `Config::MAX_BATCH_URLS` (10000), `Config::DEFAULT_BATCH_MAX_URLS`, `Config::DEFAULT_DEBOUNCE_PER_URL` (600), `Config::DEFAULT_THROTTLE_PER_MINUTE` (60), `Config::DEFAULT_HTTP_TIMEOUT` (10.0), `Config::PRODUCTION_ENVIRONMENTS` (`['prod', 'production']`), `Config::DEFAULT_MAX_URL_LENGTH`, `Config::DEFAULT_LOG_URLS`, `Config::DEFAULT_FORBIDDEN_ESCALATION`, `Config::DEFAULT_RETRY_*`, `Config::DEFAULT_RESOLVER_MAX_VIA_*`, `Config::LOG_EVENTS`. ## Одно понятие — три ключа Адаптеры разделяют ключи ядра под теми же именами и добавляют несколько своих; у некоторых понятий ключ (или набор значений) свой в каждом фреймворке. Таблицы генерируются из кода (`bin/config-table`) и проверяются в CI — см. раздел «One concept, three keys» в [английской версии](configuration.md#one-concept-three-keys). Коротко: | Понятие | Symfony | Laravel | Yii2 | |---|---|---|---| | Режим доставки `dispatch` | `auto` \| `messenger` \| `sync` \| `none` | `queue` \| `sync` \| `none` (без `auto`) | `auto` \| `queue` \| `sync` \| `none` | | Очередь / транспорт | `messenger.transport` | `queue.connection` | `queue.component` | | Локали для `locales: all` | `framework.enabled_locales` | `router.locales` | `router.languages` | | Переключатель ORM-хуков | `doctrine.enabled` | `eloquent.enabled` | `active_record.enabled` | | Маршрут файла ключа | `key_file.path` | `key_file.path` | `key_file.pattern` | | Куда логировать | `logging.channel` | `logging.channel` | `logging.category` | ## Переменные окружения `Config::fromEnv()` читает `getenv()`, слитый с `$_SERVER` и `$_ENV`. Первый аргумент — свой массив вместо них, второй — другой префикс вместо `INDEXNOW_`. Пустые строки считаются незаданными. | Переменная | Опция | |---|---| | `INDEXNOW_ENABLED` | `enabled` (любой булев литерал, который принимает `filter_var`) | | `INDEXNOW_KEY` | `key` | | `INDEXNOW_PREVIOUS_KEY` | `previous_key` | | `INDEXNOW_HOSTS` | `hosts` в виде `host=key,host2=key2`; `key_location`/`base_url` по хостам — только через `fromArray()` | | `INDEXNOW_STRICT_HOSTS` | `strict_hosts` | | `INDEXNOW_KEY_LOCATION` | `key_location` | | `INDEXNOW_BASE_URL` | `base_url` | | `INDEXNOW_ENGINES` | `engines`, через запятую (`api` или `yandex,bing`) | | `INDEXNOW_DISPATCH` | `dispatch` | | `INDEXNOW_BATCH_MAX_URLS` | `batch.max_urls` | | `INDEXNOW_DEBOUNCE_PER_URL` | `debounce.per_url` | | `INDEXNOW_THROTTLE_PER_MINUTE` | `throttle.max_requests_per_minute` | | `INDEXNOW_HTTP_TIMEOUT` | `http.timeout` | | `INDEXNOW_USER_AGENT` | `http.user_agent` | | `INDEXNOW_KEY_FILE_ENABLED` (`INDEXNOW_SERVE_KEY_FILE` всё ещё побеждает) | `key_file.enabled` | | `INDEXNOW_KEY_FILE_CACHE_MAX_AGE` | `key_file.cache_max_age` | | `INDEXNOW_DEBOUNCE_STORE` | `debounce.store` | | `INDEXNOW_HTTP_CLIENT` | `http.client` | | `INDEXNOW_DRY_RUN` | `dry_run` | | `INDEXNOW_ENV`, иначе `APP_ENV` | `environment` | | `INDEXNOW_PRODUCTION_ENVIRONMENTS` | `production_environments`, через запятую | | `INDEXNOW_MAX_URL_LENGTH` | `max_url_length` | | `INDEXNOW_LOG_URLS`, `INDEXNOW_FORBIDDEN_ESCALATION` | `logging.max_urls`, `logging.forbidden_escalation` | | `INDEXNOW_RETRY_MAX_ATTEMPTS`, `INDEXNOW_RETRY_BASE_DELAY`, `INDEXNOW_RETRY_MULTIPLIER`, `INDEXNOW_RETRY_MAX_DELAY`, `INDEXNOW_RETRY_SERVER_ERROR_DELAY` | `retry.*` | ## Хосты, ключи и `strict_hosts` Поддомены для IndexNow — отдельные хосты: каждому нужен свой файл ключа. Три схемы: - **Один сайт.** Задайте `key` и `base_url`. Каждый отправляемый хост использует этот ключ. - **Несколько сайтов, у каждого ключ.** Заполните `hosts`. Хосты вне карты всё ещё падают на `key`. - **Несколько сайтов и ничего лишнего.** Задайте `strict_hosts: true`. Ключ по умолчанию тогда применяется только к хосту `base_url`; URL любого другого неперечисленного хоста пропускаются с причиной `no_key`, а не объявляются под чужим ключом. Рекомендуется всегда, когда URL могут прийти из пользовательского ввода или мультитенантной базы. `hosts..key_location` переопределяет URL файла ключа только для этого хоста и должен быть на нём. `hosts..base_url` даёт хосту свою абсолютную базу для генерации URL вне запроса — у консольной команды или воркера очереди нет контекста запроса, и без него все сайты генерировались бы на едином `base_url`. `Config::baseUrlFor($host)` возвращает эту базу, падает на `base_url`, когда хост — базовый, и `null` иначе. Ключи перечисляются через `Config::$hosts`, `Config::$keyLocations` и `Config::$hostBaseUrls` (карты по хостам в нижнем регистре). Чтобы брать ключи из базы или реестра тенантов, реализуйте `Key\KeyProviderInterface`. ## Страховка dry-run `Config::fromArray()` сам включает `dry_run`, когда выполняется **всё** из: нет `key`, нет `hosts`, задан `environment`, и он не в `production_environments` (по умолчанию `Config::PRODUCTION_ENVIRONMENTS`). Разработчик, не задавший `INDEXNOW_KEY` локально, получает логирование вместо падения при загрузке и никогда не доходит до реального API. Обратный случай стоит алерта: `dry_run` включён при `environment` production — не отправляется ничего. `Config::isProduction()` сообщает об этом, а `Check\Checker` поднимает это как **ошибку**, а не warning. И зеркально: вне production настроенный ключ и незаданный `dry_run` — ошибка `check` (стейджинг отправлял бы боевые URL); явный `dry_run: false` говорит, что окружение отправляет нарочно, и даёт warning. ## Валидация Конструктор бросает `Exception\ConfigurationException` при: - `enabled` без `key`, `hosts` или `dry_run`; - `key` (или ключе хоста) вне `[A-Za-z0-9-]{8,128}`; - ключе `hosts`, который не голое имя хоста (есть схема, порт или путь); - `base_url`, который не абсолютный `http(s)` URL или несёт credentials; - `key_location`, который не абсолютный `http(s)` URL с путём или не на хосте `base_url` — движки принимают файл ключа только с отправляемого хоста; - `hosts..key_location` или `hosts..base_url`, указывающих на другой хост; - `batch.max_urls` вне `1..10000`, отрицательных `debounce.per_url` или `throttle.max_requests_per_minute`, `http.timeout` не больше нуля, пустом списке `engines`; - `dispatch`, который не короткий идентификатор, `http.user_agent` с переводами строк; - `strict_hosts` без единого известного хоста; - имени движка, которое не известный движок, не алиас и не `https` endpoint (простой `http` разрешён только на loopback-хостах, для mock-серверов). Каждое сообщение называет факт, допустимое и как исправить. `Config::fromArray()` дополнительно отклоняет нечисловые значения числовых опций вместо тихого дефолта. ## Производные конфигурации `with()` принимает имена аргументов конструктора и возвращает проверенную копию; неизвестное имя бросает исключение. Смена `dryRun` делает копию «явной» (`$dryRunExplicit`), прочие изменения флаг сохраняют. ```php $probe = $config->with(dryRun: false, engines: ['yandex']); $config->withDryRun(true); // сокращение $config->userAgent(); // действующая строка User-Agent $config->baseHost(); // хост base_url в нижнем регистре или null ``` ## Опечатки в конфигурации адаптера `Config::OPTIONS` перечисляет каждый ключ, который понимает `fromArray()`, в форме через точку. `Config::unknownOptions($data, $allowed)` возвращает ключи массива, которые ни опции ядра, ни в `$allowed`, так что адаптер предупреждает о `debounce.per_urls`, а не молча игнорирует. Вложенные ключи перечисляйте как `block.key`, никогда как голый `block`: голое имя останавливает проверку внутри блока. Адаптеры получают это через `Adapter\ConfigFactory::load()` (`ownedOptions:`), который также сливает дефолты адаптера, разрешает `dispatch: auto` и превращает невалидное значение в `critical`-строку лога и выключенный `Config` вместо исключения. ```php $unknown = Config::unknownOptions($userConfig, ['messenger', 'messenger.bus', 'doctrine.enabled']); if ($unknown !== []) { $logger->warning('indexnow: unknown option(s): {options}', ['options' => implode(', ', $unknown)]); } ``` Вложенные массивы проверяются на один уровень по пути через точку; `hosts` принимается всегда, потому что его ключи — имена хостов. Имя блока в `$allowed` (например `messenger`) разрешает весь блок, так что адаптер перечисляет либо имя блока, либо отдельные пути, которыми владеет. # Диагностика [English version](troubleshooting.md) Начните с `php artisan indexnow:check`, затем `php artisan indexnow:explain "App\Models\Post" `, затем канал лога на `debug`. ## Не отправляется ничего | Симптом | Причина | Исправление | |---|---|---| | `check`: `configuration: ...` и exit 1 | значение из `env()` невалидно; IndexNow работает выключенным | исправьте значение; точная ошибка напечатана | | `explain`: `when: published -> false` сразу после `create()` | у атрибута `when` дефолт только в базе, свежая модель его не имеет | `protected $attributes = ['published' => false]` на модели, или задайте явно | | `explain`: `no #[IndexNow] rule` | у модели нет атрибута и она не зарегистрирована | добавьте атрибут или `IndexNowKit::observe()` | | URL разрешены, POST нет | `dispatch: queue` и нет воркера | `php artisan queue:work` или `dispatch: sync` | | лог: `rule "..." ignores this update (fields ...)` | фильтр `fields` не совпал с изменёнными атрибутами | добавьте атрибут в `fields` или уберите фильтр | | лог: `Cannot generate route "posts.show": Missing required parameter` | `params` правила не совпадают с маршрутом | `params: ['post' => 'self']` для route model binding, либо назовите каждый параметр | | лог: `Cannot read "foo" on App\Models\Post: no method foo(), getFoo(), isFoo() or hasFoo(), no property "foo"…` | опечатка в аксессоре | исправьте аксессор; годятся атрибуты, casts, accessors, relations и методы | | массовый `update()` ничего не изменил в индексе | bulk-запросы не вызывают событий (A13) | `IndexNowKit::submitModels($query->get())` или `indexnow:submit-model` | | `attach()` на pivot ничего не изменил | операции с pivot не дают событий владельцу | `$touches = ['posts']` на связанной модели, правило без фильтра `fields` | ## Отправлено, но движок отвечает | Ответ | Смысл | Исправление | |---|---|---| | 403 (`invalid_key`, job failed) | `https:///.txt` недоступен или с другим телом | `indexnow:check`; CDN может кэшировать старый файл (`key_file.cache_max_age`) | | 422 (`unprocessable`) | URL другого хоста, чем `host`, или файл ключа на другом хосте | ключ на каждый хост (`hosts`), `strict_hosts: true` | | 429 (`rate_limited`) | слишком много запросов | job освобождается с `Retry-After`; понизьте `throttle.max_requests_per_minute` | | 202 (`pending`) | принято, проверка ключа в процессе | нормально для нового ключа; `check --live` позже ответит 200 | Счётчик 403, эскалирующий в `critical`, — на процесс: несколько воркеров считают каждый свои пять. ## Дубли, тайминги - Тот же URL не отправляется повторно в течение `debounce.per_url` (600 с). `--force` обходит это; `debounce.store: cache` делит окно между запросами и воркерами, `memory` — нет. - Всё из одного запроса уходит одним батчем на `terminating`; job, сохраняющий модели, сбрасывает после job. - Откатившаяся транзакция ничего не отправляет; откат savepoint внутри `DB::transaction()` отбрасывает только внутренние URL. ## Стейджинг отправил свои URL | Симптом | Причина | Исправление | |---|---|---| | Bing/Яндекс показывают URL `staging.example.com`, или в логе `failed` / `unprocessable` (422) по ним | стейджинг работает с боевым ключом и без `INDEXNOW_DRY_RUN`; URL сгенерированы на его хосте | вне production задайте `INDEXNOW_DRY_RUN=1` (или `INDEXNOW_ENABLED=0`); с core 0.6 `check` на такой копии падает | | стейджинг отдаёт боевой файл ключа | `key_file.enabled` включён везде | `key_file.enabled: false` вне production | | движки проиндексировали страницы стейджинга | хост стейджинга ответил `200` и отдал ключ | отдавайте `410` (или `noindex` + запрет в `robots.txt`) и ротируйте ключ, если он утёк | | preview-окружение должно отправлять нарочно | — | `INDEXNOW_DRY_RUN=0` явно; `check` тогда предупреждает, а не падает | ## Дубли с `memory` и несколькими воркерами | Симптом | Причина | Исправление | |---|---|---| | один и тот же URL отправляет каждый воркер | `debounce.store: memory` — на процесс | `debounce.store` = общий store (`cache`); `check` предупреждает о `memory` | | дубли сразу после сбоя кэша | store fails open | ожидаемо и ограничено; следите за warning `debounce store unavailable` | | дубли после деплоя | кэш сброшен или изменился `debounce.key_prefix` | безвредно один раз; держите префикс стабильным | ## Тестовые окружения Вне `production_environments` отсутствующий ключ включает `dry_run`: запросы логируются, не отправляются. `check` об этом предупреждает; в production `dry_run` — ошибка. Настроенный ключ без `INDEXNOW_DRY_RUN` вне production — ошибка `check` (стейджинг отправлял бы боевые URL). ## Где логи Канал `indexnow.logging.channel` (иначе канал по умолчанию). Уровни: успех `debug`, 202 `info`, 403/400 `error`, 422/429/5xx `warning`, невалидная конфигурация `critical`, молчаливые решения (`when` false, несовпадение `fields`) `debug`. Полный список — в [руководстве по эксплуатации ядра](../core/operations.md). # Диагностика [English version](troubleshooting.md) Начните отсюда: ```bash bin/console indexnow:check # конфигурация, файлы ключей, движки, проводка bin/console indexnow:explain 'App\Entity\Post' 42 # одна сущность, весь путь решения, ничего не отправляет ``` Вдвоём они закрывают большинство обращений без единой строки лога. ## Не отправляется ничего Идите по списку сверху вниз; каждый пункт называет улику, которая его подтверждает. | Причина | Улика | |---|---| | `enabled: false` | `skipped` / `disabled` и одна `info`-строка на вызов | | `dry_run: true` | `skipped` / `dry_run`, `info`-строка с полным телом запроса | | невалидная конфигурация выключила IndexNow | одна `critical`-строка при загрузке, `indexnow:check` завершается ненулевым кодом | | у сущности нет подходящего правила | `indexnow:explain` говорит, какое правило пропущено и почему | | коллектор ни разу не сброшен | `warning: N collected URL(s) discarded: the unit of work ended without flush()` | | хуки Doctrine не активны | `indexnow:check` печатает `doctrine: entity hooks are NOT active` | | Messenger настроен, воркер не запущен | панель профайлера показывает отправленные в очередь URL; в логе воркера пусто | **Невалидная конфигурация** — самая коварная. Плейсхолдер окружения нельзя проверить при компиляции, поэтому пустой `INDEXNOW_KEY` в production не ломает контейнер — он логирует ``` indexnow: invalid configuration, IndexNow is disabled until it is fixed: (run "bin/console indexnow:check") ``` на уровне `critical` и работает выключенным. Поставьте алерт на эту строку и включите `indexnow:check` в smoke-тесты деплоя. ## 403 — файл ключа Самая частая ошибка, и означает всегда одно: движок не смог проверить `https:///.txt`. ```bash bin/console indexnow:check --host www.example.com curl -i https://www.example.com/.txt ``` Ответ должен быть `200`, `text/plain`, тело — ровно ключ, **без редиректа**. Частые причины: - маршруты бандла не импортированы (без Flex): добавьте `@IndexNowKitBundle/config/routes.php` в `config/routes.yaml`; - `key_file.enabled: false` (или устаревший `serve_key_file: false`) без `key_location`; - CDN или reverse proxy отдаёт закэшированный старый файл после ротации ключа — `key_file.cache_max_age` по умолчанию 300 секунд именно поэтому; - catch-all маршрут или firewall перехватывает `/{key}.txt` раньше; - редиректы `http → https` или на `www` на файле ключа; - запрос пришёл на другой хост, чем тот, которому принадлежит ключ, — контроллер отвечает 404 намеренно. Пятый подряд 403 для хоста логируется один раз на `critical` с текстом *«submissions for this host are not being indexed»*. Это строка для пейджинга. Счётчик живёт в процессе: несколько воркеров считают каждый свои пять. ## 422 — unprocessable URL не принадлежат хосту, под которым отправлены, или `keyLocation` невалиден. На практике: - консольная команда или воркер сгенерировали URL на неверном хосте, потому что нет `base_url` для хоста — см. [multi-domain.md](multi-domain.md); - правило закрепило `host:`, ключ которого не настроен; - `key_location` указывает на другой хост. Слой конфигурации это отклоняет, так что такое бывает только с собственным `KeyProviderInterface`. `indexnow:explain` печатает хост и файл ключа для каждого URL, который отправил бы, — это решает вопрос сразу. ## 429 — rate limited Движок попросил сбавить темп. Результат `retryable`, `retryAfter` заполнен, если движок его назвал. - С Messenger транспорт повторяет и учитывает `Retry-After` на Symfony 7.2+. Делать ничего не нужно. - С `dispatch: sync` повторов нет: батч логируется и отбрасывается. Переходите на Messenger, если видите это регулярно. - Понизьте `throttle.max_requests_per_minute`, если причина — ваш собственный трафик. Он считается на процесс. - Обычный триггер — миграция или импорт, объявляющий десятки тысяч URL разом; делайте это порциями из воркера. ## Конкретный URL так и не отправляется ```bash bin/console indexnow:explain 'App\Entity\Post' 42 --event=updated ``` Вывод идёт по правилам по порядку и печатает для каждого подписку на событие, результат `when`, фильтр `fields` и разрешённые URL; затем для каждого URL — нормализованную форму, хост, замаскированный ключ, URL файла ключа и состояние дебаунса. Типичные находки: - **`when: false`** — сущность не опубликована, страницы нет. Правильное поведение. - **фильтр `fields`** — изменилось поле, которое правилу не важно. Добавьте его в `fields` или уберите `fields`, чтобы отправлять при любом изменении. - **не подписано** — `events` правила не включает это событие. - **`debounced`** — отправлено в последние `debounce.per_url` секунд. `--force` у `indexnow:submit` обходит это. - **`no key for host`** — хост не в `hosts` и не хост `base_url`. - **ошибки правила** — отсутствующий аксессор или маршрут, который не генерируется, печатаются красным и логируются на `error`. ## Публикация и снятие ведут себя странно `when` вычисляется до и после изменения. `true → false` отправляет удаление, чтобы движки переобошли 404; `false → true` — создание. Две вещи для проверки: - **Поле за `when` должно быть в change set.** `when: 'isPublished'` сопоставляется полю `published` по конвенции (`isPublished` → `published` → `is_published`, `getStatus` → `status`). Когда имя аксессора не связано с колонкой, назовите поля явно: `whenFields: ['status', 'visibleFrom']`. - **Удаление черновика ничего не отправляет** — намеренно. Страница никогда не была публичной. ## Панель профайлера `enabled`, `dry_run`, `dispatch` и `strict_hosts` наверху говорят о режиме. Затем: - **Collected** — URL, которые запрос буферизовал. Ноль здесь значит, что ни одно правило ничего не дало: проблема выше, в правилах или guard'ах. - **Results** — что реально отправлено, по движку и хосту, с HTTP-кодом и `reason`. Заполняется и при синхронной доставке, которая происходит после ответа. - С Messenger таблица результатов пуста намеренно, панель говорит, что HTTP-результаты в логе воркера. - **Key files** — URL, который должен отдавать каждый управляемый хост, с замаскированным ключом. Откройте один в браузере как первую проверку. - **base_url** — помечен, если не задан: тогда относительные URL и генерация в консоли или воркере не работают. ## Логи Всё идёт в Monolog-канал `indexnow`, каждое сообщение начинается с `indexnow: `. ```yaml # config/packages/dev/monolog.yaml monolog: handlers: indexnow: type: stream path: '%kernel.logs_dir%/indexnow.log' level: debug channels: ['indexnow'] ``` На время диагностики ставьте `debug`: причина, по которой правило решило **не** давать URL, логируется только на этом уровне. Все тексты и уровни — в [руководстве по эксплуатации](../core/operations.md). Ключи замаскированы в каждой строке, включая тела ответов и сообщения исключений, — эти файлы безопасно прикладывать к багрепорту. ## Стейджинг отправил свои URL | Симптом | Причина | Исправление | |---|---|---| | Bing/Яндекс показывают URL `staging.example.com`, или в логе `failed` / `unprocessable` (422) по ним | стейджинг работает с боевым ключом и без `dry_run`; URL сгенерированы на его хосте | вне production задайте `INDEXNOW_DRY_RUN=1` (или `INDEXNOW_ENABLED=0`); с core 0.6 `check` на такой копии падает | | стейджинг отдаёт боевой файл ключа | `key_file.enabled` включён везде | `key_file.enabled: false` вне production — ни один движок не сможет проверить ключ на этом хосте | | движки проиндексировали страницы стейджинга | хост стейджинга ответил на них `200` и отдал ключ | отдавайте на стейджинге `410` (или `noindex` + запрет в `robots.txt`) и ротируйте ключ, если он утёк | | preview-окружение должно отправлять нарочно | — | задайте там явный `dry_run: false`; `check` тогда предупреждает, а не падает | ## Дубли с `memory` и несколькими воркерами | Симптом | Причина | Исправление | |---|---|---| | один и тот же URL отправляет каждый воркер в течение минут | `debounce.store: memory` — на процесс; у каждого веб- и queue-воркера своё окно | `debounce.store` = общий кэш; `check` предупреждает о `memory` | | дубли сразу после сбоя кэша | store fails open: пока кэш лежит, дедупликации нет | ожидаемо и ограничено (один запрос на URL); следите за частотой warning `debounce store unavailable` | | дубли после деплоя | общий кэш сброшен или изменился `debounce.key_prefix` | безвредно один раз; держите префикс стабильным для приложения | ## Дубли после сбоя кэша Debounce store fails open. Если Redis или пул кэша недоступен, дедупликация пропускается и пишется warning (`debounce store unavailable, submitting without de-duplication`). Видимый симптом — всплеск повторных отправок, а не потерянные, и это правильный размен: дубль стоит один запрос, пропуск оставляет устаревший контент в индексе. `indexnow:check` читает тестовый ключ через настроенный пул и печатает `debounce: 600s per URL, shared through cache pool "cache.app" (RedisAdapter)` либо ошибку с именем пула, если он непригоден; `debounce.store: memory` помечается как «на процесс». ## `indexnow:sitemap` в read-only контейнере `SitemapReader` держит каждый документ во временном файле во время разбора. При `readOnlyRootFilesystem` и без записываемого `/tmp` дефолтный `sitemap.spool: auto` падает на память и пишет один warning на прогон; память ограничена `sitemap.max_bytes` на документ (50 МиБ), так что ничего не ломается, просто стоит RAM. Смонтируйте `emptyDir` на `/tmp` (или задайте `TMPDIR` / `sitemap.spool_dir` на записываемый том), чтобы вернуть временный файл, или задайте `spool: memory` явно. `indexnow:check` печатает, куда spool'ятся документы и почему temp dir непригоден. Sitemap, обрывающийся на середине («ends early (truncated download or broken sitemap)»), или потерянное соединение повторяются `sitemap.fetch_retries` раз; если всё равно не вышло, прочитанные URL отправляются, команда завершается с 1, и повторный запуск безопасен (IndexNow идемпотентен, debounce store поглощает повторы). Для ежедневного cron с `--changed-since` берите окно шире интервала (`"2 days"`), чтобы упавший прогон не оставил дыры. ## Как просить помощи Приложите вывод `bin/console indexnow:check`, релевантные строки лога `indexnow` и вывод `bin/console indexnow:explain` для одной затронутой сущности. Эти три закрывают почти каждый вопрос мейнтейнера. # Диагностика [English version](troubleshooting.md) Начните с `php yii indexnow/check`, затем `php yii indexnow/explain 'app\models\Post' `, затем категория лога `indexnow` на `debug` (`log.targets[].categories = ['indexnow']`, `levels = ['error', 'warning', 'info', 'trace']`). ## Не отправляется ничего | Симптом | Причина | Исправление | |---|---|---| | `check`: `configuration: ...` и exit 1 | значение в `options` (обычно из `getenv()`) невалидно; IndexNow работает выключенным | исправьте значение; точная ошибка напечатана и залогирована один раз на `critical` | | лог: `unknown option(s) in the indexnow configuration: ...` | опечатка в `options` (`debounce.per_urls`, `key_file.enabld`) | путь через точку называет ключ | | `explain`: `when: published -> false` сразу после `save()` | у атрибута `when` дефолт только в базе | задайте его до `save()` или вызовите `loadDefaultValues()` в `init()` | | `explain`: `no #[IndexNow] rule` | у класса нет атрибута и он не зарегистрирован | добавьте `#[IndexNow]`, `IndexNowBehavior`, `active_record.models` или `observe()` | | `explain` даёт URL, лог на save молчит | behavior не подключён, либо `active_record.enabled` / `enabled` выключены | `check` печатает `active_record: ... hooked` или причину | | `debug`-лог: `did not land on commit` | изменение откатилось savepoint'ом, или verifier не увидел строку | ожидаемо для отката; про verifier см. [commit-safety.md](commit-safety.md) | | `debug`-лог: `debounced` | URL отправлен в пределах `debounce.per_url` | `--force` у команды, или уменьшите окно | | `warning`: `skipping ... unmanaged host` | хост URL — ни `base_url`, ни в `hosts` | добавьте хост в `hosts` или исправьте `base_url` | | консоль: `base_url is not set` | URL относительные, а запроса нет | задайте `base_url`; обязателен при `dispatch: queue` | ## Файл ключа | Симптом | Причина | Исправление | |---|---|---| | `GET /.txt` — 404 | pretty URL выключены, или `key_file.enabled` false, или ключ другой | `check` печатает правило; `urlManager.enablePrettyUrl = true` | | движки отвечают 403 | отдаваемое тело — не отправляемый ключ, редирект, или закэшированный старый файл после ротации | `curl -i https://host/.txt`; `key_file.cache_max_age` — 300 с нарочно | | `check`: `key file ... returned 200`, но 403 остаётся | `hosts` и отправляемый хост различаются (www vs apex) | перечислите каждый отправляемый хост в `hosts`, включите `strict_hosts` | ## Очередь | Симптом | Причина | Исправление | |---|---|---| | `check`: `queue component "queue" is not configured` | `dispatch: queue` (или `auto`, разрешённый в него) без yii2-queue | установите `yiisoft/yii2-queue` и настройте компонент, либо `dispatch: sync` | | попытки 429/5xx идут подряд без задержки | `yii\queue\sync\Queue` игнорирует задержку повторного push | реальный драйвер (db, redis, amqp); `check` предупреждает о `SyncQueue` | | `attempts` очереди не ограничивает 429/5xx | job перепушивает их сам, до `retry.max_attempts` | задайте `retry.max_attempts` (и задержки `retry.*`) в опциях компонента, см. [queue.md](queue.md) | | `ttr` превышен на больших батчах | `queue.ttr` (300 с) мал для числа батчей | поднимите `queue.ttr` или понизьте `batch.max_urls` | ## Sitemap `php yii indexnow/sitemap --dry-run` перечисляет, что было бы отправлено; `sitemap.enabled is false.` значит, что блок выключен или невалиден (причина в логе). `check` печатает, куда spool'ятся документы; на read-only ФС задайте `sitemap.spool_dir` или `sitemap.spool: memory`. Ридер — в [`indexnowkit/sitemap`](../sitemap/index.md). ## Отправлено, но движок отвечает | Ответ | Смысл | Исправление | |---|---|---| | 403 (`invalid_key`, job rejected permanently) | `https:///.txt` недоступен или с другим телом | `indexnow/check`; CDN может кэшировать старый файл (`key_file.cache_max_age`) | | 422 (`unprocessable`) | URL другого хоста, чем `host`, или файл ключа на другом хосте | ключ на каждый хост (`hosts`), `strict_hosts: true`; консольным URL нужен `base_url` на нужном хосте | | 429 (`rate_limited`) | слишком много запросов | job перепушивает себя с `Retry-After`; понизьте `throttle.max_requests_per_minute` | | 202 (`pending`) | принято, проверка ключа в процессе | нормально для нового ключа; `check --live` позже ответит 200 | Счётчик 403, эскалирующий в `critical`, — на процесс: несколько воркеров считают каждый свои пять. ## Дубли, тайминги - Тот же URL не отправляется повторно в течение `debounce.per_url` (600 с). `--force` обходит это; `debounce.store: cache` делит окно между запросами и воркерами, `memory` — нет. - Всё из одного запроса уходит одним батчем после ответа (`Response::EVENT_AFTER_SEND`); консольная команда сбрасывает при завершении, воркер очереди — после каждого job. - Откатившаяся транзакция ничего не отправляет; изменение, перечитанное на commit и не подтверждённое строкой, отбрасывается ([commit-safety.md](commit-safety.md)). ## Стейджинг отправил свои URL | Симптом | Причина | Исправление | |---|---|---| | Bing/Яндекс показывают URL `staging.example.com`, или в логе `failed` / `unprocessable` (422) по ним | стейджинг работает с боевым ключом и без `dry_run`; URL сгенерированы на его хосте | вне production задайте `'dry_run' => true` (или `'enabled' => false`); с core 0.6 `check` на такой копии падает | | стейджинг отдаёт боевой файл ключа | `key_file.enabled` включён везде | `key_file.enabled: false` вне production | | движки проиндексировали страницы стейджинга | хост стейджинга ответил `200` и отдал ключ | отдавайте `410` (или `noindex` + запрет в `robots.txt`) и ротируйте ключ, если он утёк | | preview-окружение должно отправлять нарочно | — | явный `'dry_run' => false`; `check` тогда предупреждает, а не падает | ## Дубли с `memory` и несколькими воркерами | Симптом | Причина | Исправление | |---|---|---| | один и тот же URL отправляет каждый воркер | `debounce.store: memory` — на процесс | `debounce.store` = компонент кэша (`cache`); `check` предупреждает о `memory` | | дубли сразу после сбоя кэша | store fails open | ожидаемо и ограничено; следите за warning `debounce store unavailable` | | дубли после деплоя | кэш сброшен или изменился `debounce.key_prefix` | безвредно один раз; держите префикс стабильным |