{"schema_version":"1.0","content_hash":"sha256:d813faed511725250a72671ebfbd9341a566a05a8e07b4797d52e94ebdd581f5","updated_at":"2026-06-30","title":"Getting CTF Data from WRDS","summary":"How to pull the three CTF input tables from WRDS with Python or R, and write them to the Parquet files a submission receives.","source":{"html":"https://jkpfactors.com/ctf/dataset-access","markdown":"https://jkpfactors.com/ctf/dataset-access.md","json":"https://jkpfactors.com/ctf/dataset-access.json"},"body_markdown":"## Overview\n\nThis guide shows how to use Python or R to get data for the Common Task Framework-inspired competition proposed by [Hellum, Jensen, Kelly, and Pedersen (2025)](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5242901).\n\n> **Note**\n>\n> We'll extract the following three tables from the WRDS database:\n>\n> - `contrib_global_factor.ctff_features`\n> - `contrib_global_factor.ctff_chars`\n> - `contrib_global_factor.ctff_daily_ret`\n\n## Prerequisites\n\n#### General\n\n- A **[WRDS](https://wrds-www.wharton.upenn.edu/)** account with access to:\n\n  - **[CRSP Monthly Stock File](https://wrds-www.wharton.upenn.edu/pages/get-data/center-research-security-prices-crsp/annual-update/stock-security-files/monthly-stock-file/)**\n  - **[CRSP Daily Stock File](https://wrds-www.wharton.upenn.edu/pages/get-data/center-research-security-prices-crsp/annual-update/stock-security-files/daily-stock-file/)**\n  - **[Compustat North America](https://wrds-www.wharton.upenn.edu/pages/get-data/compustat-capital-iq-standard-poors/compustat/north-america-daily/)**\n\n#### Tool-specific setup\n\n**Python**\n\n1. Install packages (uv/pip/conda all fine):\n\n   ```bash\n   uv add pandas sqlalchemy psycopg2-binary keyring\n   ```\n2. Store WRDS password securely (replace `WRDS_USERNAME` and `WRDS_PASSWORD`):\n\n   ```python\n   import keyring\n   keyring.set_password(\"wrds\", \"WRDS_USERNAME\", \"WRDS_PASSWORD\")\n   ```\n\n**R**\n\n1. Install packages:\n\n   ```r\n   install.packages(c(\"DBI\", \"RPostgres\", \"keyring\"))\n   ```\n2. Store WRDS password securely (replace `WRDS_USERNAME`):\n\n   ```r\n   keyring::key_set(service = \"wrds\", username = \"WRDS_USERNAME\")\n   ```\n\n## Data download\n\n**Python**\n\n```python\nimport keyring\nimport pandas as pd\nfrom sqlalchemy import create_engine, text\nfrom sqlalchemy.engine import URL\n\n# --- Credentials from OS keychain (keyring) ---\ncreds = keyring.get_credential(\"wrds\", None)\nif creds is None:\n    raise RuntimeError(\n        \"\"\"No WRDS credentials stored.\n        Run: keyring.set_password('wrds', 'WRDS_USERNAME', 'WRDS_PASSWORD').\"\"\"\n    )\nwrds_un, wrds_pw = creds.username, creds.password\n\n# --- WRDS Postgres connection (SSL required) ---\n# Use URL.create() to properly handle special characters in passwords\nurl = URL.create(\n    drivername=\"postgresql+psycopg2\",\n    username=wrds_un,\n    password=wrds_pw,\n    host=\"wrds-pgdata.wharton.upenn.edu\",\n    port=9737,\n    database=\"wrds\",\n    query={\"sslmode\": \"require\"},\n)\nengine = create_engine(url)\n\n# --- Helper: simple fetch ---\ndef wrds_fetch(sql: str) -> pd.DataFrame:\n    with engine.connect() as conn:\n        return pd.read_sql_query(text(sql), conn)\n\n# --- Download tables ---\nctff_features  = wrds_fetch(\"SELECT * FROM contrib_global_factor.ctff_features;\")\nctff_chars     = wrds_fetch(\"SELECT * FROM contrib_global_factor.ctff_chars;\")\nctff_daily_ret = wrds_fetch(\"SELECT * FROM contrib_global_factor.ctff_daily_ret;\")\n\n# --- Save locally ---\n# For example, we use (requires pyarrow package):\n#    ctff_features.to_parquet(\"data/raw/ctff_features.parquet\", index=False)\n#    ctff_chars.to_parquet(\"data/raw/ctff_chars.parquet\", index=False)\n#    ctff_daily_ret.to_parquet(\"data/raw/ctff_daily_ret.parquet\", index=False)\n```\n\n> **Note**\n>\n> **Memory issues?** If memory is tight for, e.g., `ctff_chars`, fetch in chunks:\n>\n> ```python\n> with engine.connect() as conn:\n>     parts = pd.read_sql_query(\n>         text(\"SELECT * FROM contrib_global_factor.ctff_chars;\"),\n>         conn,\n>         chunksize=500_000,\n>     )\n>     ctff_chars = pd.concat(parts, ignore_index=True)\n> ```\n\n**R**\n\n```r\n# --- Libraries ---\nlibrary(keyring)\nlibrary(DBI)\nlibrary(RPostgres)\n\n# --- Credentials from OS keychain (keyring) ---\nif (nrow(key_list(\"wrds\"))==0) {\n  stop(\n    \"No WRDS credentials stored.\\n\",\n    \"Run: keyring::key_set(service = 'wrds', username = 'WRDS_USERNAME').\"\n  )\n}\nwrds_un <- key_list(service = \"wrds\")$username[1]\nwrds_pw <- key_get(\"wrds\", wrds_un)\n\n# --- Connect to WRDS (SSL required) ---\ncon <- dbConnect(\n  RPostgres::Postgres(),\n  host    = \"wrds-pgdata.wharton.upenn.edu\",\n  port    = 9737,\n  dbname  = \"wrds\",\n  sslmode = \"require\",\n  user    = wrds_un,\n  password = wrds_pw\n)\non.exit(dbDisconnect(con), add = TRUE)\n\n# --- Helper: simple fetch ---\nwrds_fetch <- function(sql) DBI::dbGetQuery(con, sql)\n\n# --- Download tables ---\nctff_features  <- wrds_fetch(\"SELECT * FROM contrib_global_factor.ctff_features;\")\nctff_chars     <- wrds_fetch(\"SELECT * FROM contrib_global_factor.ctff_chars;\")\nctff_daily_ret <- wrds_fetch(\"SELECT * FROM contrib_global_factor.ctff_daily_ret;\")\n\n# --- Save locally ---\n# For example, we use (requires arrow package):\n#   arrow::write_parquet(ctff_features, \"data/raw/ctff_features.parquet\")\n#   arrow::write_parquet(ctff_chars, \"data/raw/ctff_chars.parquet\")\n#   arrow::write_parquet(ctff_daily_ret, \"data/raw/ctff_daily_ret.parquet\")\n```","body_blocks":[{"type":"heading","level":2,"text":"Overview"},{"type":"paragraph","text":"This guide shows how to use Python or R to get data for the Common Task Framework-inspired competition proposed by [Hellum, Jensen, Kelly, and Pedersen (2025)](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5242901)."},{"type":"note","variant":"note","title":"Note","blocks":[{"type":"paragraph","text":"We'll extract the following three tables from the WRDS database:"},{"type":"list","style":"unordered","items":["`contrib_global_factor.ctff_features`","`contrib_global_factor.ctff_chars`","`contrib_global_factor.ctff_daily_ret`"]}]},{"type":"heading","level":2,"text":"Prerequisites"},{"type":"heading","level":4,"text":"General"},{"type":"list","style":"unordered","items":[{"text":"A **[WRDS](https://wrds-www.wharton.upenn.edu/)** account with access to:","blocks":[{"type":"list","style":"unordered","items":["**[CRSP Monthly Stock File](https://wrds-www.wharton.upenn.edu/pages/get-data/center-research-security-prices-crsp/annual-update/stock-security-files/monthly-stock-file/)**","**[CRSP Daily Stock File](https://wrds-www.wharton.upenn.edu/pages/get-data/center-research-security-prices-crsp/annual-update/stock-security-files/daily-stock-file/)**","**[Compustat North America](https://wrds-www.wharton.upenn.edu/pages/get-data/compustat-capital-iq-standard-poors/compustat/north-america-daily/)**"]}]}]},{"type":"heading","level":4,"text":"Tool-specific setup"},{"type":"tabs","key":"tabset-1-1","tabs":[{"id":"python","pane":"tabset-1-1","label":"Python","blocks":[{"type":"list","style":"ordered","items":[{"blocks":[{"type":"paragraph","text":"Install packages (uv/pip/conda all fine):"},{"type":"code","language":"bash","code":"uv add pandas sqlalchemy psycopg2-binary keyring","copy":true}]},{"blocks":[{"type":"paragraph","text":"Store WRDS password securely (replace `WRDS_USERNAME` and `WRDS_PASSWORD`):"},{"type":"code","language":"python","code":"import keyring\nkeyring.set_password(\"wrds\", \"WRDS_USERNAME\", \"WRDS_PASSWORD\")","copy":true}]}]}],"default":true},{"id":"r","pane":"tabset-1-2","label":"R","blocks":[{"type":"list","style":"ordered","items":[{"blocks":[{"type":"paragraph","text":"Install packages:"},{"type":"code","language":"r","code":"install.packages(c(\"DBI\", \"RPostgres\", \"keyring\"))","copy":true}]},{"blocks":[{"type":"paragraph","text":"Store WRDS password securely (replace `WRDS_USERNAME`):"},{"type":"code","language":"r","code":"keyring::key_set(service = \"wrds\", username = \"WRDS_USERNAME\")","copy":true}]}]}]}]},{"type":"heading","level":2,"text":"Data download"},{"type":"tabs","key":"tabset-2-1","tabs":[{"id":"python","pane":"tabset-2-1","label":"Python","blocks":[{"type":"code","language":"python","code":"import keyring\nimport pandas as pd\nfrom sqlalchemy import create_engine, text\nfrom sqlalchemy.engine import URL\n\n# --- Credentials from OS keychain (keyring) ---\ncreds = keyring.get_credential(\"wrds\", None)\nif creds is None:\n    raise RuntimeError(\n        \"\"\"No WRDS credentials stored.\n        Run: keyring.set_password('wrds', 'WRDS_USERNAME', 'WRDS_PASSWORD').\"\"\"\n    )\nwrds_un, wrds_pw = creds.username, creds.password\n\n# --- WRDS Postgres connection (SSL required) ---\n# Use URL.create() to properly handle special characters in passwords\nurl = URL.create(\n    drivername=\"postgresql+psycopg2\",\n    username=wrds_un,\n    password=wrds_pw,\n    host=\"wrds-pgdata.wharton.upenn.edu\",\n    port=9737,\n    database=\"wrds\",\n    query={\"sslmode\": \"require\"},\n)\nengine = create_engine(url)\n\n# --- Helper: simple fetch ---\ndef wrds_fetch(sql: str) -> pd.DataFrame:\n    with engine.connect() as conn:\n        return pd.read_sql_query(text(sql), conn)\n\n# --- Download tables ---\nctff_features  = wrds_fetch(\"SELECT * FROM contrib_global_factor.ctff_features;\")\nctff_chars     = wrds_fetch(\"SELECT * FROM contrib_global_factor.ctff_chars;\")\nctff_daily_ret = wrds_fetch(\"SELECT * FROM contrib_global_factor.ctff_daily_ret;\")\n\n# --- Save locally ---\n# For example, we use (requires pyarrow package):\n#    ctff_features.to_parquet(\"data/raw/ctff_features.parquet\", index=False)\n#    ctff_chars.to_parquet(\"data/raw/ctff_chars.parquet\", index=False)\n#    ctff_daily_ret.to_parquet(\"data/raw/ctff_daily_ret.parquet\", index=False)","copy":true},{"type":"note","variant":"note","title":"Note","blocks":[{"type":"paragraph","text":"**Memory issues?** If memory is tight for, e.g., `ctff_chars`, fetch in chunks:"},{"type":"code","language":"python","code":"with engine.connect() as conn:\n    parts = pd.read_sql_query(\n        text(\"SELECT * FROM contrib_global_factor.ctff_chars;\"),\n        conn,\n        chunksize=500_000,\n    )\n    ctff_chars = pd.concat(parts, ignore_index=True)","copy":true}]}],"default":true},{"id":"r","pane":"tabset-2-2","label":"R","blocks":[{"type":"code","language":"r","code":"# --- Libraries ---\nlibrary(keyring)\nlibrary(DBI)\nlibrary(RPostgres)\n\n# --- Credentials from OS keychain (keyring) ---\nif (nrow(key_list(\"wrds\"))==0) {\n  stop(\n    \"No WRDS credentials stored.\\n\",\n    \"Run: keyring::key_set(service = 'wrds', username = 'WRDS_USERNAME').\"\n  )\n}\nwrds_un <- key_list(service = \"wrds\")$username[1]\nwrds_pw <- key_get(\"wrds\", wrds_un)\n\n# --- Connect to WRDS (SSL required) ---\ncon <- dbConnect(\n  RPostgres::Postgres(),\n  host    = \"wrds-pgdata.wharton.upenn.edu\",\n  port    = 9737,\n  dbname  = \"wrds\",\n  sslmode = \"require\",\n  user    = wrds_un,\n  password = wrds_pw\n)\non.exit(dbDisconnect(con), add = TRUE)\n\n# --- Helper: simple fetch ---\nwrds_fetch <- function(sql) DBI::dbGetQuery(con, sql)\n\n# --- Download tables ---\nctff_features  <- wrds_fetch(\"SELECT * FROM contrib_global_factor.ctff_features;\")\nctff_chars     <- wrds_fetch(\"SELECT * FROM contrib_global_factor.ctff_chars;\")\nctff_daily_ret <- wrds_fetch(\"SELECT * FROM contrib_global_factor.ctff_daily_ret;\")\n\n# --- Save locally ---\n# For example, we use (requires arrow package):\n#   arrow::write_parquet(ctff_features, \"data/raw/ctff_features.parquet\")\n#   arrow::write_parquet(ctff_chars, \"data/raw/ctff_chars.parquet\")\n#   arrow::write_parquet(ctff_daily_ret, \"data/raw/ctff_daily_ret.parquet\")","copy":true}]}]}],"code_samples":[{"language":"bash","section":"Tool-specific setup","variant":"Python","step":1,"code":"uv add pandas sqlalchemy psycopg2-binary keyring"},{"language":"python","section":"Tool-specific setup","variant":"Python","step":2,"code":"import keyring\nkeyring.set_password(\"wrds\", \"WRDS_USERNAME\", \"WRDS_PASSWORD\")"},{"language":"r","section":"Tool-specific setup","variant":"R","step":1,"code":"install.packages(c(\"DBI\", \"RPostgres\", \"keyring\"))"},{"language":"r","section":"Tool-specific setup","variant":"R","step":2,"code":"keyring::key_set(service = \"wrds\", username = \"WRDS_USERNAME\")"},{"language":"python","section":"Data download","variant":"Python","code":"import keyring\nimport pandas as pd\nfrom sqlalchemy import create_engine, text\nfrom sqlalchemy.engine import URL\n\n# --- Credentials from OS keychain (keyring) ---\ncreds = keyring.get_credential(\"wrds\", None)\nif creds is None:\n    raise RuntimeError(\n        \"\"\"No WRDS credentials stored.\n        Run: keyring.set_password('wrds', 'WRDS_USERNAME', 'WRDS_PASSWORD').\"\"\"\n    )\nwrds_un, wrds_pw = creds.username, creds.password\n\n# --- WRDS Postgres connection (SSL required) ---\n# Use URL.create() to properly handle special characters in passwords\nurl = URL.create(\n    drivername=\"postgresql+psycopg2\",\n    username=wrds_un,\n    password=wrds_pw,\n    host=\"wrds-pgdata.wharton.upenn.edu\",\n    port=9737,\n    database=\"wrds\",\n    query={\"sslmode\": \"require\"},\n)\nengine = create_engine(url)\n\n# --- Helper: simple fetch ---\ndef wrds_fetch(sql: str) -> pd.DataFrame:\n    with engine.connect() as conn:\n        return pd.read_sql_query(text(sql), conn)\n\n# --- Download tables ---\nctff_features  = wrds_fetch(\"SELECT * FROM contrib_global_factor.ctff_features;\")\nctff_chars     = wrds_fetch(\"SELECT * FROM contrib_global_factor.ctff_chars;\")\nctff_daily_ret = wrds_fetch(\"SELECT * FROM contrib_global_factor.ctff_daily_ret;\")\n\n# --- Save locally ---\n# For example, we use (requires pyarrow package):\n#    ctff_features.to_parquet(\"data/raw/ctff_features.parquet\", index=False)\n#    ctff_chars.to_parquet(\"data/raw/ctff_chars.parquet\", index=False)\n#    ctff_daily_ret.to_parquet(\"data/raw/ctff_daily_ret.parquet\", index=False)"},{"language":"python","section":"Data download","variant":"Python","code":"with engine.connect() as conn:\n    parts = pd.read_sql_query(\n        text(\"SELECT * FROM contrib_global_factor.ctff_chars;\"),\n        conn,\n        chunksize=500_000,\n    )\n    ctff_chars = pd.concat(parts, ignore_index=True)"},{"language":"r","section":"Data download","variant":"R","code":"# --- Libraries ---\nlibrary(keyring)\nlibrary(DBI)\nlibrary(RPostgres)\n\n# --- Credentials from OS keychain (keyring) ---\nif (nrow(key_list(\"wrds\"))==0) {\n  stop(\n    \"No WRDS credentials stored.\\n\",\n    \"Run: keyring::key_set(service = 'wrds', username = 'WRDS_USERNAME').\"\n  )\n}\nwrds_un <- key_list(service = \"wrds\")$username[1]\nwrds_pw <- key_get(\"wrds\", wrds_un)\n\n# --- Connect to WRDS (SSL required) ---\ncon <- dbConnect(\n  RPostgres::Postgres(),\n  host    = \"wrds-pgdata.wharton.upenn.edu\",\n  port    = 9737,\n  dbname  = \"wrds\",\n  sslmode = \"require\",\n  user    = wrds_un,\n  password = wrds_pw\n)\non.exit(dbDisconnect(con), add = TRUE)\n\n# --- Helper: simple fetch ---\nwrds_fetch <- function(sql) DBI::dbGetQuery(con, sql)\n\n# --- Download tables ---\nctff_features  <- wrds_fetch(\"SELECT * FROM contrib_global_factor.ctff_features;\")\nctff_chars     <- wrds_fetch(\"SELECT * FROM contrib_global_factor.ctff_chars;\")\nctff_daily_ret <- wrds_fetch(\"SELECT * FROM contrib_global_factor.ctff_daily_ret;\")\n\n# --- Save locally ---\n# For example, we use (requires arrow package):\n#   arrow::write_parquet(ctff_features, \"data/raw/ctff_features.parquet\")\n#   arrow::write_parquet(ctff_chars, \"data/raw/ctff_chars.parquet\")\n#   arrow::write_parquet(ctff_daily_ret, \"data/raw/ctff_daily_ret.parquet\")"}]}