shanhui notes

applied ai (product) engineer, i usually write about building applied AI systems, learning hardware, and random musings about the changing nature of knowledge work

building a AI jobs search engine with Exa search APIs

Sat 08 August 2026 · 4955 words · 33 min read

to build a search engine requires many components and algorithms, however the 2 critical components are the search service and the indexer + crawlers. i have built several crawlers and indexes before for different use cases such as Chinese news articles and eCommerce product research, so i figured it would be an interesting learning exercise to focus on the search component for a weekend project idea: “to search all AI related jobs in APAC region”. it needs to be comprehensive and efficient to search. comprehensive means that the indexer needs to crawl and aggregate all jobs from various sources that are related to AI which is not that simple.

some companies like OpenAI, Exa, and Cursor are definite AI-native companies, so their engineering open jobs should be included. but you also have big tech MNCs like Google, Bytedance, Alibaba, and Microsoft that have subset of their jobs related to AI, but not all. then their are lesser known Chinese and Singaporean robotic startups that need “rust embedded engineers” or Singaporean government institutes like NTU that need ML researchers. So there are various definitions of what an AI job role is, and the system can not just search by keywords, it needs to actually understand the meaning by classification of the job post content with some steering from prompts evals and enrichment classifiers to generate tags and category filters.

the problem of keyword search

the central problem that all search engines must address is can you return a relevant set of candidates based on a user query f(Q) = []C ranked by relevancy that satisfies the user’s intent?

and can you do it at decent speed and cost that will not bankrupt the service providers at scale. most job search engines are quite bad at this because it does not understand semantic meaning, instead it matches by keywords or filters.

if you search "fde robotics, sg", it will probably not return anything because the job post might contain “Forward Deployed Engineer”, “Singapore”, and “humanoid robots”, but not "fde robotics, sg startup". and if you search in Malay or Chinese "人工智能应用工程师", it will not work if the job post is in English (unless you translate the job posting at insert time). while technologies like Postgres FTS is great and goes a long way, but there are limitations that get in the way of how human users naturally search for jobs.

what about more nuance queries like "applied ai engineer, prefer rust, no degree required", or "heavy industries/manufacturer hiring jr ai engineer team for digital transformation" (good prospects to market enterprise AI agents solutions to) how would a search engine even begin to classify such queries? behind every search are intents, and the engine needs to surface that intent behind the text string to return relevant results to the user. these intents could be a job search or they could be enterprise sales engineer prospecting new clients.

in order to tackle this problem, we have to reorganize content in the higher dimensions of vectors and the distance between these vectors. instead of keywords, vectors are encoded in N dimensions and embeds the meanings of the chuck of text. then you can compare how similar 2 vectors are using various similarity algorithms like cosine distance, which compares the angle of the vectors, are they directionally similar concepts?

a note on vector databases

for less than 500k vectors (our expected upper limit since we will have a sweeper for expired links and dead links), pgvector is a fine choice and pgvector implements the HNSW (hierarchical navigable small world) index which for most purposes is good enough and matches the commercial options like Pinecone and Qdrant, plus Postgres is opensource and proven to be super reliable. during research for this project, i read that Exa implemented their own vector database in Rust that supports binary quantization, support Matroyoska embeddings, and SIMD. if you want to index the entire web, then that is what you have to do and i am sure would be super interesting to work on. for a small/medium project like this, pgvector will do just fine.

HNSW algorithm for vector search

Hierarchical Navigable Small World graphs are super efficient indexes for vector similarity search. HNSW works by creating a multi-layered graph, where the entry point is the query vector. it searches for vectors with the closest distance to the query vector within within a single layer until the search reaches a local minimum, then the search moves to the next layer and repeats the nearest neighbors search. once the search reaches the local minimum of layer 0, then the search stops. that is the gist of it, but you can watch this video for a more complete explanation. more on the parameter options for creating the HNSW index in Postgres below.

embeddings models vs. large language models

although i am not an ai research engineer or pretend to understand all the mathematics of model pre-training, i still find it valuable to develop a basic understanding of the underlying models used in what i build. similar to large language models (LLMs), embedding models are based on the transformer architecture. embeddings models are encoder only vs LLMs that have a decoder. the attention mechanism for embedding models are bidirectional, meaning every token can see other tokens vs. LLMs that can only see the tokens before it. the way embedding models are trained is with many pairs of query and documents, then a similarity matrix is computed over all QxD pairs. softmax is applied over all documents so that the final vector values are a N-dimension unit sphere. the core difference between a embeddings model LLM is that the embeddings model like X and Y.

fundamentally, embedding models are trained to pull related texts together and unrelated text apart, but they have no concept of next token prediction like LLMs so they can not generate new content. embedding models use the same attention formula, but without the MASK, it sees all the tokens, forwards and backwards:

Attention(Q, K, V) = softmax(QKᵀ / √d + MASK) × V

embedding models are typically also 100x to 1000x smaller than LLMs and perform better at recall of related texts than LLMs that are 100x larger in size. this is because embedding models use bidirectional attention to create an holistic representation of the entire input. every token’s encoding is attended towards every other token. LLMs also have this ability as an side effect of their training process to predict the next token, but far less efficient than pure embedding models.

the embeddings model i choose is the BGE-M3 which is a 568M parameter model that was trained on ~1B query/document text pairs. the variant we will be using is the DeepInfra multilingual BGE-M3 embeddings model. another one that i looked at was the intfloat/multilingual-e5-large model also available on DeepInfra, however the maximum tokens for this model is 512 until the document gets truncated so i went with the BGE-M3 instead which handles documents up to 8192 tokens. a typical job post might have between 500 to 2000 tokens so going with the BGE-M3 means we will not need to create multiple chunks from each posting to embed. both models have multilingual support for 100+ languages which includes all the ones we care about in APAC (English, Chinese, Malay, Vietnamese, Thai, …). this means that queries can be in any of those languages and the search should still return relevant results back.

general architecture of the search engine

three parts: - parsers/ - collects listings from ATS and career pages (Ashby, Exa, direct sources, SG career sites, markdown files) -> normalizes into a common job post schema (Rust) - jobsapi, Rust Axum API + embed indexed worker + pgvector search - webapp/ client web implemented using Rust/Yew/WASM because why not?

the parsers aggregate the web’s job posts into a centralized table store job_listings, then we have a embeddings indexer that runs every 30 seconds that creates a source text with job title, company name, meta data, and post content artifact, creates an 1024dim embedding using the BGE-M3 model and stores the embeddings in a job_embeddings table. the entire system is decoupled and works async. if the web crawler suddenly indexes 1000 new job posts, the embeddings indexer still indexes at a set pace of every X seconds to spread out the load throughout the day. and embedding that fails will be picked up again at the next interval, and persistent failures will be marked and ignored.

search control flow, embeddings + vector search

when users search for "rust engineer, embedded systems, Singapore" in the webapp, we encode the query and send to the search api.

pub async fn search(query_text: &str) -> Result<Vec<JobListingResponse>, JobsapiError> {
  let url = format!("{}/search?query={}", jobsapi_url(), urlencoding(query_text));
  //...
  resp.json().await.map_err(JobsapiError:Network)
}

GET /search?query=rust%20engineer,%20embedded%20systems,%20singapore

after basic validations, we send it to the job_service.search_embeddings(query, limit) which uses the same embeddings model BAAI/bge-m3-multi on DeepInfra to generate the queries embedding.

POST api.deepinfra.com/v1/openai/embeddings
  -> [0.1, 0.3, ...] (1024-dim vector)  ~150ms

then we compare and search this query vector with the vectors we have indexed in our job_embeddings table.

let out = self.embed_service.generate_embeddings(&[query], InputType::Query).await?;
let vecq = pgvector::Vector::fromP(out[0].embeddings.clone());
let rows = self.repo.search_by_embeddings(vecq, limit).await?;
// ...

SELECT jl.id, jl.company_name, jl.job_title, ...
  (jeb.dense_embedding <=> $1) AS distance
FROM job_embeddings jeb
JOIN job_listings jl ON jl.id = jeb.job_listing_id
ORDER BY distance
LIMIT $2
;

CREATE INDEX idx_job_embeddings_hnsw
  ON job_embeddings
  USING hnsw (dense_embedding vector_cosine_ops)
  WITH (m = 16, ef_construction = 200)

the search returns a set of job listing results and the distance score which we can convert into a similarity percent for the frontend ((2.0 - distance) / 2.0 * 100.0).round(). we return around 50 results which we will rerank in the next step. if we return the results now without first re-ranking, the results would capture related job posts and may be better than pure keyword text search, but it will not be the most relevant.

the reason for this is because BGE-M3 bi-encodes the query and document independently, compares with cosine similarity. the comparison is fast, but some information is loss during this process. both the query and the document vector must summarize the intent into 1 1024-dim vector.

another issue is that a low cosine distance between query and document does not mean user relevance. cosine distance simply measures how close 2 vectors are in geometric space. one weak spot would be negation queries like “rust engineer, NO crypto” which often fails for pure embedding vector search systems and end up returning crypto job postings. hard filters like “Singapore”, “min: 150k” become soft constraints in vector embedding space. another challenge of job postings is that many job postings are similar with repeated templated sections, sure we can filter out these sections, but string matching and classification is not always accurate for all edge cases. this creates many candidates with similar distances, which is why we need to add a reranker model as the 2nd stage of our search engine.

optimization #1: adding Qwen3 reranker model to improve search relevancy

the second stage of the search involves the addition of a reranker model. i choose the Qwen/Qwen3-Reranker-4B model for this component which is also available on DeepInfra at https://api.deepinfra.com/v1/inference.

Stage 1: BGE-M3 → pgvector → top 2N candidates (~200ms)
Stage 2: Qwen-Reranker → re-sort → top N candidates (~2500ms)

the important thing to know about rerank models is that it does not touch or see the embeddings vectors, they operate on the full text tokens, in our case the combined job posting. the vector search is used to recall a subset of document candidates using a similarity comparisons like cosine or euclidean similarity, then you would send all the documents along with the plain text query with instructions to the rerank model to return relevancy scores. this means that the set of recall documents from vector search need to be greater than then the final set you want to return to the client webapp for display because you want to have a higher probability that relevant documents will be within the set to be rerank. so you might have 50 documents you want recalled, and then rerank them, and return the top 25 final results.

the rerank process is a couple orders of magnitude more costly in dollar and latency than the vector search so you want to be conscious of costs at scale. i estimated that using DeepInfra inference token prices, it will cost around $1.40/1k searches or $1.4k/1M. the levers that will affect cost will be the size of the model, there are 0.6B/4B/8B versions for the same Qwen reranker model, and the length of the document you send to be rerank. if the average job posting is 2k tokens, you can truncate the length to 1k and save around half the cost. keep in mind that if you recall N documents, you need to send all 50 to be rerank, so you can adjust the N also, but that will affect relevancy if too low.

because of costs, we need to make sure proper rate limiting is in place per IP address. we set it to 10 req / min / IP for the rerank route GET /search?query=...&rerank=true.

the reranker POST request looks something like this:

{
  "queries": ["user query..."],
  "documents": ["doc1...", "doc2...", ...],
  "instruction": "custom instructions for comparison..."
    // Given a job search query, rank job postings by relevance to the candidate's intent
}

the RerankerService also handles - skipping rerank if less than 10 candidates returned from vector search - truncate each document to the first 4k chars, which means the beginning of documents get seen and ranked. the job requirements and company information are usually at the beginning of the posting with boilerplate info about equal opportunities policies and benefits at the end which is less relevant as a search signal. - it logs the cost and tokens usage so we can monitor and aggregate it if we want to get track of costs at scale. - it logs duration of the rerank request in milliseconds for p99 monitoring - we return only the subset, eg. top 25 ranked results - we have a is_transient help fn that marks HTTP errors, empty responses, and 401/403/429/5xx as retryable.

for a successful rerank request we then:

for each candidate i: (row_i, score_i)
sort by score_i desc
take 25
score in response = clamp(relevance, 0..1)

optimization #2: caching the queries embedding vector

in our current flow, the user’s search query gets converted to a query vector every time, even for common queries like “Applied AI Engineer”. this embeddings request takes a couple hundred milliseconds and the cost can be material if there are millions of such queries per day.

   User searches "rust engineer singapore"
     → POST api.deepinfra.com/v1/openai/embeddings   (~250ms, $)
     → BGE-M3 encodes query → 1024-dim vector
     → pgvector scans candidates

most search follow a pareto distribution, the most popular 20% of queries will account for 80% or more of searches.

the generated vectors can be cached because the same query string (with instructions prefix) always produces the same resulting embedding vectors. it is a pure function and DETERMINISTIC process. so, encode('rust engineer, singapore') is a pure function that always creates the same vector when the same embeddings model is used making the operation mathematically sound.

there are 2 choices to serve as the data store for the search query vectors. the first is an in memory cache, and second is a Postgres table. generally in memory is 10x faster (basically instant, ~1µs) than on disk and fetch from disk is 10-20x faster than from network (without any cache).

in our case, we want search trends analytics so that we can optimize the search even more later. for this reason, going with Postgres cache table is the right choice here. plus we can hot load the most popular hits from the search cache to create a mini in memory cache if we want to speed things up at the edge.

the next step is to design the cache key and normalization. user search queries can come in all shapes in forms: “rust engineer”, “Rust Engineer”, “rust, engineer” should all collapse into the same normalize query key in our cache.

// basic normalization removes whitespace and lowercase
fn normalize_query(raw: &str) -> String {
  raw.split_whitespace()
    .collect::<Vec<_>>()
    .join(" ")
    .to_lowercase()
}

if the embeddings model might be switched in the future, you might want to prefix the cache key with the model name format!("{model_name}:{normalized_query}")

the Postgres search_queries table would look something like this:

CREATE TABLE search_queries (
    query_key    TEXT PRIMARY KEY,        -- normalized
    query_text   TEXT NOT NULL,           -- for analytics / debugging
    dense_embeddings vector(1024) NOT NULL,
    model_name   TEXT NOT NULL,  
    hits         BIGINT NOT NULL DEFAULT 0,
    last_used    TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    created_at   TIMESTAMPTZ NOT NULL DEFAULT NOW()
)

with Postgres, we get to see the most popular queries trending analytics for free (useful for product development):

SELECT query_text, hits FROM search_queries ORDER BY hits DESC LIMIT 20;

some other benefits of storing in Postgres - no evictions needed, millions of rows are fine; in memory cache you might want to evict after 10k entries - survives server restarts - more columns can be added for analytics, more flexibility query using SQL, e.g most popular searches last 7 days in Malaysia

the cache lookup flow:

   GET /search?query="rust engineer singapore"
   │
   ├─ normalize → key = "bge-m3-multi:rust engineer singapore"
   │
   ├─ SELECT dense_embedding FROM search_queries WHERE query_key = $1
   │   │
   │   ├─ HIT:  → vector straight to pgvector scan    (~2ms)
   │   │          UPDATE hits = hits + 1
   │   │
   │   └─ MISS: → POST DeepInfra embeddings          (~250ms)
   │              INSERT INTO query_embeddings ...                                                  
   │              → vector to pgvector scan
   │
   └─ pgvector search (unchanged) (~10ms)

pub struct QueryCacheService {
  pool: PgPool,
  embed_service: EmbeddingsService,
}

impl QueryCacheService {
  pub async fn get_cached_or_generate(&self, query: &str) -> Result<pgvector::Vector, JobEmbedError> {
    let key = cache_key(&query);

    // 1. try cache first
    if let Some(v) = self.fetch(&key).await? {
      self.increase_hits(&key).await?;
      tracing::debug!(query, cache = "hit", "query embeddings served from cache");
      return Ok(v);
    }

    // 2. miss, pay the api call
    tracing::debug!(query, cache = "miss", "query embedding computed");
    let out = self.embed_service.generate_embeddings(&[query], InputType::Query).await?;
    let vec = pgvector::Vector::from(out[0].embeddings.clone());

    // 3. store the generated vector with race condition safe insert
    sqlx::query("INSERT INTO query_embeddings (cache_key, model_name, input_type, normalized_query, dense_embedding)
      VALUES ($1, $2, 'query', $3, $4)
      ON CONFLICT (cache_key) DO NOTHING")
    .bind(&key).bind(model).bind(query).bind(&vec).execute(&self.pool).await?;

    Ok(vec)
  }
}

with this queries cache table in place, we can load the top 100 or so search queries into memory on a periodic basis to keep most popular queries always hot:

SELECT query_key FROM search_queries ORDER BY hits DESC LIMIT 100;

the latency saving are 10x when cache is hit, and the estimate cost savings at scale are around 80% by the implementation of a queries cache.

more optimisation that might be worth doing

a. hybrid keyword + vector search

there are 2 variant of this which i will touch upon in brief. the first 1 is using Postgres FTS to_tsvector which will add another column with GIN index to the job_embeds table.

ts_rank is in [0, 1] so we need to come up with a weight formula to fuse the vector similarity ranking and keyword ts_rank together, for instance 0.7 * normalized(vec_score) + 0.3 keyword_score.

the other fusion algorithm that came up in my research is RRF (Reciprocal Rank Fusion) which i am told is the modern SOTA and what Elasticsearch uses. it is calculated as:

RRF(d) = Σ [ 1 / (k + rank_i(d)) ]

RRF does not using the original similarity scores at all and relies only on the rank from both sets.

one caveat of using Postgres FTS is that it only works with English text ts_vector('english', '机器学习') will be tokenized as one unit so if the documents include multi-lingual content, then the results will be off.

b. BGE-M3 native embedding sparse vectors

this path i need to research more, but certain hybrid embedding models like the BGE-M3 natively supports both dense and sparse vector embeddings. we use the dense vectors for the standard vector cosine search, however the sparse vectors can be used as a keyword signal. the sparse vector can be computed in the same api request as the dense vectors and stored into another column. the benefit of this approach is that it supports multilingual search, but weaker than real FTS because it uses bpe tokens, not whole words.

c. set filters for hard columns (easy)

this is your standard WHERE pre-filters for hard columns like city, region, remote_ok, visa_provided, salary min and max. pre-filtering on metadata narrows the candidate set that HNSW needs to scan through making the search faster and more relevant. hard filters combine well with vector search because vectors are “fuzzy” approximations that are good for locating higher dimension

d. using LLM to enrich tags and metadata

we could using a lighter, low cost, open model like the Deepseek-V4-Flash to scan job postings that are missing metadata columns like tags, location information, or benefits. not all metadata can be captured during the deterministic parsing process, so an LLM can fill into the missing gaps in information and the embeddings worker can regenerate the vector embeddings for more relevant search. at scale, enriching a thousand listings would only cost a couple dollars with open models like deepseek-v4-flash or another classifier model.

search quality evals

perhaps one of the most important steps is to create proper evals so you can measure improvements and to catch degradations is search relevancy. the 2 metrics that are often used to measure relevancy. the industry standard is NDCG@K (Normalized Discounted Cumulative Gain at K) which measure the most relevant results at the top.

  DCG@K = Σ(relevance_i / log2(i+1))  for i=1..K
  NDCG@K = DCG@K / ideal_DCG@K

another older metric that can be used to measure is MRR (Mean Reciprocal Rank) MRR = avg(1 / rank_first_relevant) across all queries, although some practitioner have recommended against using MRR for modern search.

but in either case, in order to use any search metric we have to prepare an eval set which is a manual process that takes time and careful consideration.

   query,job_id,relevance
   "rust engineer singapore",abc-123,3
   "AI researcher japan",def-111,3
   "devops AWS docker",def-222,1
   ...

one method that AI can be used to save time is using “LLMs as a Judge” to bootstrap the eval dataset with “Humans in the Loop” for quality verification of the LLM synthesize dataset:

System: "Score this job listing's relevance to the query on a 0-3 scale.
         Query: {query}
         Job: {title} at {company}. {description}"

the eval iteration pipeline looks something like this:

  1. make search algo and index changes (tweak weights, add reranker)
  2. run eval harness
  3. compare metrics NDCG@K and MRR
  4. if there is a drop, regression -> investigate
  5. repeat

one interesting thing i want to try in the future is Andrei Karpathy’s autoresearch to automate the optimisation of search engine relevancy once the metrics and harness is set up.

Exa for job content parsing

Exa is a search api startup that is trying to build perfect search for humans and agents. you can search for queries like “rust hardware engineers in Singapore that have a active dev blog” and get a list of matches for the data fields you care about. to bootstrap the database for this project, i used Exa’s search api to find AI-native companies like Cursor, Cognition, Sierra, and Exa themselves that were hiring in the APAC region.

then i created a parser using Exa’s content api that scrapes the contents of job posts. with the Exa Content API, you can create a structured json struct and a prompt to extract exactly the fields you want from a page. it does not work all the time, but it is an easy way to bootstrap an universal parser without manually writing one for each job source.

  summary: Some(ExaSummaryOption::Config(ExaSummaryConfig {
      query: Some(
          "Extract structured job posting fields from this career page. For tags, return 3–8 lowercase department/product categories and required languages/frameworks only — never locations, amenities, or benefits.\n\nNever extract form fields, form questions, or labels from the form".into(),
      ),
      schema: Some(job_post_summary_schema()),
  })),

    serde_json::json!({
        "type": "object",
        "required": ["company_name", "job_title", "city_location"],
        "additionalProperties": false,
        "properties": {
            "company_name": { "type": "string", "description": "Hiring company name" },
            "job_title": { "type": "string", "description": "Job title" },
            "city_location": { "type": "string", "description": "Primary city for the role" },
            "region_location": {
                "type": "string",
                "description": "Region, state, or country if mentioned; omit if unknown"
            },
            "visa_sponsorship": {
                "type": "boolean",
                "description": "True if the posting mentions visa sponsorship"
            },
            "remote_ok": {
                "type": "boolean",
                "description": "True only if the role is fully remote (not hybrid or on-site)"
            },
            "hybrid_work": {
                "type": "boolean",
                "description": "True if the role is hybrid (mix of office and remote); false if fully remote or fully on-site"
            },
            "salary_min_ks": {
                "type": "integer",
                "description": "Minimum base salary in thousands (e.g. 120 for 120000); omit if unknown"
            },
            "salary_max_ks": {
                "type": "integer",
                "description": "Maximum base salary in thousands; omit if unknown"
            },
            "currency_code": {
                "type": "string",
                "description": "ISO currency code such as USD or SGD; omit if unknown"
            },
            "offers_equity": {
                "type": "boolean",
                "description": "True if equity / stock / RSU is offered"
            },
            "offers_bonus": {
                "type": "boolean",
                "description": "True if a cash bonus is mentioned"
            },
            "tags": {
                "type": "array",
                "minItems": 3,
                "maxItems": 8,
                "items": { "type": "string" },
                "description": "3–8 lowercase tags focused only on department/product categories (e.g. engineering, ai, growth) and required languages/frameworks (e.g. python, rust, react). Do not include locations, amenities, benefits, soft skills, or workplace perks."
            }
        }
    })

and then we can map it to a common job struct:

#[derive(Debug, Clone, PartialEq)]
pub struct JobPostParsedCommon {
    pub company_name: String,
    pub job_title: String,
    pub city_location: String,
    pub region_location: Option<String>,
    pub visa_sponsorship: bool,
    pub remote_ok: bool,
    pub hybrid_work: bool,
    pub salary_min_ks: Option<i32>,
    pub salary_max_ks: Option<i32>,
    pub currency_code: Option<String>,
    pub offers_equity: bool,
    pub offers_bonus: bool,
    pub tags: Vec<String>,
    pub source_url: String,
    pub published_date: Option<String>,
    pub expiry_date: Option<String>,
    pub job_type: Option<JobType>,
    pub markdown_content: String,
    pub html_content: String,
}

the idea is that we can use Exa to seed the initial dataset and then create custom parsers for each ATS or jobs platform as we scale to save on API costs. there are also other neat Exa APIs like monitors, and agentic search that i will need to find some feature we can use for.

i wrote in more details about the embeddings indexer in this blog post.

deployment on Railway and Netlify

the deployment of the full api service, embeddings indexers, and postgres database is surprisingly simple for a full stack rust system. the jobsapi is containerized using Docker, then deployed using Railway cloud. the build pretty much just works as long as you set all the env variables the api service needs. the postgres instance also uses Railway. i wanted to minimize latency between the api service and the database as much as possible, and since Railway does not charge extra for Postgres, you pay the same compute and memory fees that the replicas use so it scale up and down depending on traffic. for the minimum amount of traffic at launch, it works out to be a few dollars per month for both the service and database. i use Railway for almost all my side projects and ventures and it’s been no issues so far. for the initial seed data, we pg_dump the local DB and pg_restore into the production postgres database on Railway using a direct pgsql connection.

the frontend webapp is deployed on Netlify. this is a couple configs that need to be added because you have to build the Rust code into WASM first so it can be distributed to browsers and run. the easiest way to do this is using a netlify.toml file. we also need to ensure that SPA routes get redirect back to the index.html. the config file tells Netlify to run the ./netlify-build.sh command.

# Monorepo: build/publish the Trunk webapp; SPA fallback for /jobs/:slug.
[build]
  base = "webapp"
  command = "./netlify-build.sh"
  publish = "dist"

# Client-side routes (yew-router) → serve index.html.
[[redirects]]
  from = "/*"
  to = "/index.html"
  status = 200

#!/usr/bin/env bash
# Netlify build: install Rust toolchain + Trunk, then release-build the WASM app.
set -euo pipefail

if [[ ! -x "$HOME/.cargo/bin/rustup" ]] && ! command -v rustup >/dev/null 2>&1; then
  curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y --default-toolchain stable
fi

# shellcheck source=/dev/null
source "$HOME/.cargo/env"

# Netlify may ship rustup shims with no default toolchain selected.
rustup default stable
rustup target add wasm32-unknown-unknown
cargo install trunk --locked

npm ci
npm run build

final notes and learnings

building a search engine for any niche vertical is a iterative process and research project with considerations to deep domain context and knowledge. it is not something you throw together once and call it shipped. you can ship a minimal viable version, but you need to have an evaluation plan on how to improve the search relevancy with real usage data over time. AI tools like Cursor and Pi makes many parts of the flow quicker and enable solo developers or small teams to iterate quickly like i did with technologies that i was new to, but experience engineering judgement and taste is what filters out the good ideas that scale from the all the noise.

the project is free and opensource here so you have iterate on it for your own business use case or reach out if you have some customisation needs to make it fit into your companies’ workflow.

i personally learned a lot and will continue trying to make the search relevancy better and add more features to it. all the improvements will be open-sourced so anyone can learn from it or contribute. you can try out the live version here: https://jobsai.shanhui.dev/