Want to extract data from web pages without managing infrastructure? Minifetch is a hosted data extraction API built on this library. Perfect for AI Agents and SEO research:
npm install minifetch-api
🎉 Sign up for an account & get free credits to start 🎉
Now accepting credit cards as well as x402 USDC payments on Base + Solana.
---
## **Includes:**
- meta tags
- hreflang
- favicons
- citations, per the Google Scholar spec
- [Open Graph Protocol (og:) Tags](http://ogp.me/)
- [Twitter Card Tags](https://developer.twitter.com/en/docs/twitter-for-websites/cards/overview/markup)
- [JSON-LD](https://moz.com/blog/json-ld-for-beginners)
- h1-h6 tags
- img tags
- relevant response headers & status code
- automatic charset detection & decoding (optional)
- the full response body as a string of html (optional)
- [x402](https://www.x402.org/) "payment required" support
v5.1.0+ Protects against:
- Infinite redirect loops
- SSRF attacks via [request-filtering-agent](https://www.npmjs.com/package/request-filtering-agent) in Node.js v18+ (custom options available)
More details below. To report a bug or request a feature please open an issue or pull request in [GitHub](https://github.com/laurengarcia/url-metadata). Please read the `Troubleshooting` section below *before* filing a bug.
## Install
Works with Node.js versions `>=6.0.0` or in the browser when bundled with Webpack (see `/example-typescript`) or Vite (see `/example-vite`) in the github repo. For Next.js, see `/example-nextjs`.
```
npm install url-metadata --save
```
## Usage
In your project file:
```javascript
const urlMetadata = require('url-metadata');
(async function () {
try {
const url = 'https://www.npmjs.com/package/url-metadata';
const metadata = await urlMetadata(url);
console.log(metadata);
} catch (err) {
console.log(err);
// Optional: handle x402 "payment required" responses
if (err.paymentRequired && err.x402) {
// Handle x402 payment details
}
}
})();
```
### Options & Defaults
To override the default options, pass in a second options argument. The default options are the values below.
```javascript
const options = {
// Customize the default request headers:
requestHeaders: {
'User-Agent': 'url-metadata (+https://www.npmjs.com/package/url-metadata)',
From: 'example@example.com'
},
// (Node.js v18+ only)
// To prevent SSRF attacks, the default option below blocks
// requests to private network & reserved IP addresses via:
// https://www.npmjs.com/package/request-filtering-agent
// Browser security policies prevent SSRF automatically.
requestFilteringAgentOptions: undefined,
// (Node.js v6+ only)
// Pass in your own custom `agent` to override the
// built-in request filtering agent above
// https://www.npmjs.com/package/node-fetch/v/2.7.0#custom-agent
agent: undefined,
// (Browser only) `fetch` API cache setting
cache: 'no-cache',
// (Browser only) `fetch` API mode (ex: 'cors', 'same-origin', etc)
mode: 'cors',
// Maximum redirects in request chain, defaults to 10
maxRedirects: 10,
// `fetch` timeout in milliseconds, default is 10 seconds
timeout: 10000,
// (Node.js v6+ only) max size of response in bytes (uncompressed)
// Default set to 0 to disable max size
size: 0,
// (Node.js v6+ only) compression defaults to true
// Support gzip/deflate content encoding, set `false` to disable
compress: true,
// Charset to decode response with (ex: 'auto', 'utf-8', 'EUC-JP')
// defaults to auto-detect in `Content-Type` header or meta tag
// if none found, default `auto` option falls back to `utf-8`
// override by passing in charset here (ex: 'windows-1251'):
decode: 'auto',
// Number of characters to truncate description to
descriptionLength: 750,
// Force image urls in selected tags to use https,
// valid for images & favicons with full paths
ensureSecureImageRequest: true,
// Include raw response body as string
includeResponseBody: false,
// Alternate use-case: pass in `Response` object here to be parsed
// see example below
parseResponseObject: undefined
};
// Basic options usage
try {
const url = 'https://www.npmjs.com/package/url-metadata';
const metadata = await urlMetadata(url, options);
console.log(metadata);
} catch (err) {
console.log(err);
// Optional: handle x402 "payment required" responses
if (err.paymentRequired && err.x402) {
// Handle x402 payment details
}
}
// Alternate use-case: parse a Response object instead
try {
// fetch the url in your own code
const response = await fetch('https://www.npmjs.com/package/url-metadata');
// ... do other stuff with it...
// pass the `response` object to be parsed for its metadata
const metadata = await urlMetadata(null, {
parseResponseObject: response
});
console.log(metadata);
} catch (err) {
console.log(err);
}
// Similarly, if you have a string of html you can create
// a response object and pass the html string into it.
const html = `