Publishing a page puts it on your server. It does not put it in the index. Between those two events sits a discovery process most businesses never look at — and in Philadelphia, where a working website is often older than the person running it, that process is usually blocked by something nobody remembers building.
The complaint arrives in the same form every time. A firm invests in twenty new pages. Three months later traffic has not moved, everyone assumes a ranking problem, and the rewriting starts. The actual situation is usually less flattering: fourteen of the twenty pages were never crawled, four were crawled and dropped, and two are indexed but competing against a page the company forgot it published in 2016.
Ranking is a contest among pages Google already knows about. If your page is not in that set, no amount of optimization applies to it. This article covers how discovery works, what quietly eats it, how the Indexing Hub in the Semalt panel intervenes, and the specific shape the problem takes on the kind of long-lived domain that is common across this region.
Existing and being found are two different states
Google finds URLs by following links from pages it already crawls, by reading sitemaps, and by direct submission. Each route has a different speed and failure mode, and a site relying on one is fragile in a way that stays invisible until a launch underperforms.
- Internal links. The primary route, and the one most damaged by rebuilds. A page reachable only through a filter or a search box is effectively unlinked.
- External links. Fast and reliable, but you do not control when they appear or where they point.
- Sitemaps. An explicit inventory. Cheap to maintain, routinely stale, and often still listing URLs removed two redesigns ago.
- Direct submission. The only route you control end to end, and the subject of most of this article.
A page can be published, correct, well written and completely undiscoverable. This happens constantly on sites that grew by accretion: a section added to a template that was never updated to link it, or a page reachable only from a menu that renders after a script runs. It exists for a visitor holding the URL and for nothing else.
Crawl budget, and what consumes it before your new pages do
Google allocates a finite amount of crawling per site, roughly a function of how large and important the site appears and how well the server responds. Nobody outside Google sees the number. What matters is that the allocation gets spent on something, and on an older site it is rarely spent on what you would choose.
| Consumer | Typical origin | Effect on discovery |
|---|---|---|
| Duplicate URL paths | Platform migration that kept both schemes alive | The same content crawled repeatedly under different addresses |
| Parameter combinations | Filters, sorting, session or tracking parameters | Near-infinite URL space generated from a handful of pages |
| Abandoned subdomains | An old shop, a campaign microsite, a staging server left public | Crawl spent on a property nobody maintains |
| Soft 404 pages | Templates returning 200 for content that no longer exists | Empty pages consuming budget and diluting quality signals |
| Slow server responses | Legacy hosting, unoptimized queries, heavy plugins | Fewer pages fetched per visit, across the whole site |
| Redirect chains | Successive rebuilds, each redirecting to the previous scheme | Several requests consumed to reach one document |
The last row recurs on Philadelphia domains more than any other. A site rebuilt in 2011, again in 2017 and again last year frequently carries all three schemes, chained. The crawler asks for the original address, is pointed at the second, then the third, and arrives having spent three requests on one document. Multiply that across a few thousand legacy URLs and much of the allocation is burned on navigation history.
Scale sets how much this matters. A neighborhood restaurant with eleven pages has no crawl budget problem and never will; if its pages are missing, the cause is discovery or quality. A hospital system with forty thousand provider pages, a university with two decades of departmental subsites, or a distributor with a filterable catalog has a real allocation problem, and the fix is subtraction before submission.
Old sites carry old mistakes
Here is what makes crawl work in this region different in character rather than degree. The problem is rarely volume. It is archaeology. Many businesses here are second- or third-generation, and their domains have been through two or three rebuilds, at least one platform migration and often a change of ownership. Nothing was ever removed, because removal felt risky and nobody could say what depended on what.
The residue is predictable. Take a family construction firm with a 2004 domain: an original static site under one path scheme, a 2013 rebuild that moved everything and left the old paths responding, a blog on a subdomain that stopped publishing in 2015, a microsite for a product line the company no longer carries, and a staging copy never password-protected. All of it is being crawled. Some of it outranks the current site.
Two URL schemes, both live
A migration that added the new structure without retiring the old one.
- Same content at two addresses, splitting signals
- Often the older path is the one ranking
Abandoned subdomains
An old blog, a defunct shop, a staging server nobody locked down.
- Crawled as a separate property
- Frequently indexed alongside production
Pages outranking their replacements
An old page with a decade of links beating the current one for its own term.
- The oldest and most valuable asset on the domain
- Never delete before you have read its query data
Pages nobody can account for
Output of a plugin, an import, or an agency relationship that ended years ago.
- Usually thin, sometimes thousands of URLs
- The cheapest budget recovered on the whole site
The change of ownership produces its own failure. A practice is acquired, a restaurant changes hands, a trade business passes to the next generation. The domain comes along, and with it content addressed to a business that no longer operates the way it did. Those pages still rank, still collect impressions, and still describe services nobody sells. Not strictly a crawl problem, but you will find it during a crawl audit.
Keep, redirect, or let go
Every legacy URL resolves to exactly one of three decisions. The work is deciding which, and the mistake almost everyone makes is deciding by intuition instead of by data. An old page that feels obsolete may be carrying the links that hold up a whole section.
| Decision | Applies when | Action | Risk if wrong |
|---|---|---|---|
| Keep | The page ranks, earns impressions, or holds external links | Update it in place, keep the URL | Leaving outdated claims live |
| Redirect | A direct successor exists on the current site | One-step 301 to the closest equivalent | Chains, or redirecting everything to the homepage |
| Let go | No traffic, no links, no successor | Return 410 or 404 and remove from sitemaps | Discarding a page with quiet inbound links |
| Consolidate | Several pages cover one topic thinly | Merge into one, redirect the rest to it | Losing detail the thin pages actually held |
Redirect discipline is where most cleanups fail, and two rules cover nearly everything. Every redirect must be one hop: point the 2011 URL directly at the current address, not at the 2017 URL that points there. And every redirect must land on the closest genuine equivalent. Mass-redirecting retired URLs to the homepage is the common shortcut, and it converts specific signals into noise.
Letting go is the step people avoid and usually the most valuable. A 410 tells Google the page is intentionally gone, a clearer instruction than a 404, and removes the URL from rotation faster. A few thousand thin URLs retired this way frees a real share of a mid-sized site's allocation, and the pages you care about get visited more often — without submitting anything.
The sitemap as a discovery instrument, not a formality
A sitemap is usually treated as a compliance artifact: generate it once, submit it, forget it. Used properly it is the cheapest discovery instrument available, and the sitemap submission tool is built around that use — it is the one place you state explicitly what the site consists of and when each part changed.
Recursive parsing to three levels
Upload a file or point at a URL; the parser follows index files down through the structure and collects what it finds.
- Three levels of depth. An index of indexes of sitemaps is parsed all the way down, which matches how large sites are actually organized.
- Up to 1,000 sitemaps per job. Enough for a hospital system or a university without splitting the work by hand.
- Two jobs at once, twenty queued. Concurrency is deliberately limited so a large parse does not starve everything behind it.
- File or URL. Submit a generated file directly, or let the panel fetch the live sitemap on the domain.
The three-level structure makes a sitemap useful for diagnosis rather than declaration. Split the inventory by section — services, locations, providers, articles, catalog — and each becomes independently countable. When one section stops being discovered you know which one, and before the traffic report says so. A flat file of eleven thousand URLs tells you only that eleven thousand URLs exist.
Direct submission, and the arithmetic of a daily budget
Beyond sitemaps sits explicit submission: a list of URLs handed to the crawler with a request to visit. The URL tracker in the Indexing Hub handles this per account, with a daily budget of 1,000 URLs and bulk submission of up to 10,000 per batch, transmitted through the IndexNow API, which serves GoogleBot and BingBot.
The two figures work together rather than contradicting each other. The batch is the queue you load; the daily budget is the rate at which it drains. A batch of 10,000 URLs is ten days of submission — a scheduling fact worth knowing before a migration weekend rather than during one.
Per-URL record, not a batch average
Every submitted URL carries its own log, which is what makes the difference between submission and indexing observable.
- Bot visit with a timestamp. Whether a crawler actually arrived, and when — the single most useful line in the log.
- Status and error detail. A failure names its cause instead of leaving you to guess at the server.
- Live counters. Submitted, discovered and failed, updated as the batch drains rather than at the end.
- Both major crawlers. IndexNow submission reaches GoogleBot and BingBot from one action.
Priority follows directly from the daily ceiling. With a thousand URLs a day, the order of the queue is a real decision rather than a formality.
Pages that earn or have moved
Commercial pages produce the inquiries, and changed pages are actively misrepresented while a stale version is cached.
- Services, locations, provider and menu pages
- Anything whose prices, hours or staff changed
Pages already settled
Indexed, unchanged URLs gain nothing from resubmission and displace something that would.
- Stable pages already ranking
- Anything you are about to edit again
What a batch actually tells you
The three live counters — submitted, discovered, failed — are only useful as ratios, and each pattern points somewhere specific. Reading them takes a minute and saves weeks of misdirected work.
- High submitted, high discovered. The mechanism is working. If traffic still has not moved, the problem is downstream: quality, intent match, or competition.
- High submitted, low discovered. Crawlers are not arriving. Look at server response times, robots directives and whether the URLs are linked from anywhere.
- Failures clustered in one section. Almost always structural — a broken template, a path that 500s, a section behind authentication.
- Failures scattered evenly. Usually the server under load, or intermittent hosting. A crawl problem with an infrastructure cause.
The pattern to watch on a legacy domain is discovery succeeding while nothing changes in Search Console. That is the strongest available evidence that pages are being crawled and judged not worth indexing — which sends you back to the content, or to the duplicate standing where your new page wants to be.
Running the numbers on an actual site
Take a mid-sized professional services firm in the collar counties with 6,000 URLs on a domain registered in 2006. An audit finds 1,900 URLs from two previous rebuilds still responding, 700 parameter variants generated by a filter, an abandoned subdomain carrying 400 pages, and 3,000 URLs that represent the current business.
The cleanup is not exotic. The 700 parameter variants are excluded from crawling. The 400 subdomain pages are reviewed: 40 redirected into the main site, 360 returning 410. Of the 1,900 legacy URLs, 250 have impressions in the last ninety days and get one-hop redirects, and the remaining 1,650 are retired. Crawlable inventory drops from 6,000 to roughly 3,290, and the pages the firm cares about become a far larger share of whatever allocation it has.
Submission comes last and is the smallest part. At 1,000 URLs a day the 3,290 survivors take four days, and most do not need submitting at all — only the changed and the new ones do. That is the proportion worth taking away: the cleanup did most of the work, while the step everyone thinks of as "the indexing tool" was four days of unattended queue.
Order matters as much as arithmetic. Redirects and removals go live first, sitemaps are regenerated to match the surviving structure, and only then is anything submitted. Submitting first asks crawlers to visit URLs you are about to change, spending the daily budget on work you will redo. The same sequence applies to a migration: publish, redirect, regenerate, submit. Our services overview covers running that on a live site without a traffic gap.
Common questions
How long after submission should a page be crawled?
Usually within a few days for a site with normal crawl activity, longer for a small or rarely updated domain. The URL log records the bot visit with a timestamp, so you do not have to infer it. If a fortnight passes with no visit recorded, the issue is on the site — response times, robots directives, or a page nothing links to.
The URL shows as discovered but does not appear in Search Console. Why?
Because discovery and indexing are different events. A crawler came and read the page; Google then decided the page did not merit inclusion. The usual causes are duplication with an existing URL, thin content, or a page that is a near-copy of a template. This is the distinction the warning above is about.
Should I submit every page on the site?
No. Submit what is new and what has changed materially. Resubmitting stable, already-indexed pages spends the daily budget without accomplishing anything, and on a 1,000-per-day ceiling that displacement is the whole cost.
We have an old subdomain nobody maintains. Delete it or redirect it?
Check its data first. If it holds pages with impressions or external links, redirect those individually into the main site and retire the rest. If it holds nothing of value — a staging copy, a dead microsite — take it offline entirely. Leaving it live is the one option with no upside.
Does IndexNow work for Google?
Submission through the API in the Indexing Hub reaches GoogleBot and BingBot, and the per-URL log shows which bot visited and when. What it cannot do is oblige either engine to index the page, which remains their decision on their criteria.
The discipline this rewards is unglamorous. Most of the value in crawl work is subtraction — retiring what should not exist, collapsing chains into single hops, turning a sitemap into an accurate statement of what the site is. Submission is the last and easiest step, and it works best on a site already cleaned up. On an old Philadelphia domain that cleanup is genuinely archaeological, worth doing once, carefully, with the query data open beside you.
Running that work in one place is the point of the unified panel: the Indexing Hub sits beside Search Console analytics and SERP tracking, so the query data deciding keep, redirect or let go is in the same interface as the tool that submits the result. Campaign automation runs $149 per month per domain for AutoSEO and $500 for FullSEO, which adds human review of on-site changes and a team of specialists; first measurable movement typically appears after four to eight weeks. Open the dashboard and run a sitemap job to see what your domain contains. Further technical reading sits in our blog archive.