Karen Markey

19 papers Journal 14Unranked 4
YearRankTypeTitle / Venue / Authors
2014 — book
Designing Online Information Literacy Games Students Want to Play.
Karen Markey, Chris Leeder, Soo Young Rieh
2012 J jnl
A Faceted Taxonomy for Rating Student Bibliographies in an Online Information Literacy Game.
Coll. Res. Libr.
Chris Leeder, Karen Markey, Elizabeth Yakel
2012 J jnl
Amount of invested mental effort (AIME) in online searching.
Inf. Process. Manag.
Soo Young Rieh, Yong-Mi Kim, Karen Markey
2012 J jnl
Through a Game Darkly: Student Experiences with the Technology of the Library Research Process.
Libr. Hi Tech
Karen Markey, Chris Leeder, Soo Young Rieh
2011 — conf
The effect of scoring and feedback mechanisms in an online educational game.
ASIST
Karen Markey, Chris Leeder
2011 J jnl
Unheard Voices: Institutional Repository End-Users.
Coll. Res. Libr.
Beth St. Jean, Soo Young Rieh, Elizabeth Yakel, Karen Markey
2010 — conf
Conceptualizing institutional repositories: using co-discovery to uncover mental models.
IIiX
Soo Young Rieh, Ji Yeon Yang, Elizabeth Yakel, Karen Markey
2010 J jnl
The Benefits of Integrating an Information Literacy Skills Game into Academic Coursework: A Preliminary Evaluation.
D Lib Mag.
Karen Markey, Fritz Swanson, Chris Leeder, Brian J. Jennings, Beth St. Jean, Victor Rosenberg, Soo Young Rieh, Robert L. Frost, Loyd Mbabu, Andrew Calvetti, Gregory R. Peters Jr., Geoffrey V. Carter, Averill Packard
2009 — conf
Institutional repositories: What's the use?
ASIST
Beth St. Jean, Soo Young Rieh, Elizabeth Yakel, Karen Markey, Raya Samet
2008 J jnl
Designing and testing a web-based board game for teaching information literacy skills and concepts.
Libr. Hi Tech
Karen Markey, Fritz Swanson, Andrea Jenkins, Brian J. Jennings, Beth St. Jean, Victor Rosenberg, Xingxing Yao, Robert L. Frost
2008 J jnl
Perceptions and Experiences of Staff in the Planning and Implementation of Institutional Repositories.
Libr. Trends
Soo Young Rieh, Beth St. Jean, Elizabeth Yakel, Karen Markey, Jihyun Kim
2008 J jnl
The Effectiveness of a Web-based Board Game for Teaching Undergraduate Students Information Literacy Concepts and Skills.
D Lib Mag.
Karen Markey, Fritz Swanson, Andrea Jenkins, Brian J. Jennings, Beth St. Jean, Victor Rosenberg, Xingxing Yao, Robert L. Frost
2008 — conf
Toward successful institutional repositories: Listening to IR staff's experiences.
ASIST
Beth St. Jean, Soo Young Rieh, Karen Markey, Elizabeth Yakel, Jihyun Kim
2007 J jnl
Census of Institutional Repositories in the U.S.: A Comparison Across Institutions at Different Stages of IR Development.
D Lib Mag.
Soo Young Rieh, Karen Markey, Beth St. Jean, Elizabeth Yakel, Jihyun Kim
2007 J jnl
Nationwide Census of Institutional Repositories: Preliminary Findings.
J. Digit. Inf.
Karen Markey, Beth St. Jean, Soo Young Rieh, Elizabeth Yakel, Jihyun Kim, Yong-Mi Kim
2007 J jnl
The Online Library Catalog: Paradise Lost and Paradise Regained?
D Lib Mag.
Karen Markey
2007 J jnl
Twenty-five years of end-user searching, Part 1: Research findings.
J. Assoc. Inf. Sci. Technol.
Karen Markey
2007 J jnl
Twenty-five years of end-user searching, Part 2: Future research directions.
J. Assoc. Inf. Sci. Technol.
Karen Markey
1981 J jnl
Levels of question formulation in negotiation of information need during the online presearch interview: A proposed model.
Inf. Process. Manag.
Karen Markey
notebooks/redb-clickhouse-TEST.ipynb
← Index notebooks/redb-clickhouse-TEST.ipynb plaintext
Truncated preview: showing the first 16384 bytes out of 5988848 total.
{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Definitions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import clickhouse_connect"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "ch_TEST_client = clickhouse_connect.get_client(host='e.rationaledge.io', port=443, username='default', password='werioubTGwvf0-dsbv_', database='redb', verify=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "ch_TEST_BETA_client = clickhouse_connect.get_client(host='k.rationaledge.io', port=443, username='default', password='werioubTGwvf0-dsbv_', database='default', verify=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<bound method HttpClient.close_connections of <clickhouse_connect.driver.httpclient.HttpClient object at 0x105fd64d0>>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ch_TEST_BETA_client.close_connections"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<clickhouse_connect.driver.summary.QuerySummary at 0x11f900810>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ch_TEST_BETA_client.command(\"\"\"\n",
    "RENAME DATABASE redb TO orig_redb;\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## DB Creation (schema 2025-08-02)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ch_TEST_client.command(\"\"\"\n",
    "CREATE DATABASE IF NOT EXISTS redb;\n",
    "\"\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define General Tables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### CREATE FULL REDB DATABASE\n",
    "def create_redb_with_all_tables(client):\n",
    "    client.command(\"\"\"\n",
    "        CREATE DATABASE IF NOT EXISTS redb\n",
    "    \"\"\")\n",
    "    create_all_general_tables(client)\n",
    "    create_all_pe_tables(client)\n",
    "\n",
    "#### CREATE FULL TABLES\n",
    "def create_all_general_tables(client):\n",
    "    create_basic_properties_dbtable(client)\n",
    "    create_hashes_dbtable(client)\n",
    "    create_capa_dbtable(client)\n",
    "    create_die_dbtable(client)\n",
    "    create_extra_findings_dbtable(client)\n",
    "\n",
    "\n",
    "##################################\n",
    "#### CREATE INDIVIDUAL TABLES ####\n",
    "##################################\n",
    "\n",
    "#### GENERAL TABLES ####\n",
    "# All tables have SETTINGS index_granularity = 8192\n",
    "def create_basic_properties_dbtable(client):\n",
    "    # Create samples_metadata table\n",
    "    client.command(\"\"\"\n",
    "        CREATE TABLE IF NOT EXISTS redb_basic_properties (\n",
    "            sha256 FixedString(64), -- primary key\n",
    "            md5 FixedString(32),\n",
    "            sha1 FixedString(40),\n",
    "            filename Nullable(String),\n",
    "            sample_name String,\n",
    "            filesize UInt64,\n",
    "            file_entropy Float64,\n",
    "            filetype LowCardinality(String),\n",
    "            filetype_mime LowCardinality(String),\n",
    "            filetype_magika LowCardinality(String),\n",
    "            analysis_date DateTime64(3, 'UTC'),\n",
    "        ) ENGINE = ReplacingMergeTree(analysis_date)\n",
    "        PRIMARY KEY (sha256)\n",
    "        ORDER BY (sha256)\n",
    "    \"\"\")\n",
    "\n",
    "def create_hashes_dbtable(client):\n",
    "    # Create hashes table\n",
    "    client.command(\"\"\"\n",
    "        CREATE TABLE IF NOT EXISTS redb_hashes (\n",
    "            sha256 FixedString(64), -- primary key\n",
    "            md5 FixedString(32),\n",
    "            sha1 FixedString(40),\n",
    "            ssdeep_hash Nullable(String),\n",
    "            tlsh_hash Nullable(String),\n",
    "            authentihash Nullable(String),\n",
    "            imphash Nullable(String),\n",
    "            impfuzzy Nullable(String),\n",
    "            typerefhash Nullable(String),\n",
    "            richhash Nullable(String),\n",
    "            richpe_hash Nullable(String),\n",
    "            richpv_hash Nullable(String),\n",
    "            richpv_hash_sorted Nullable(String),\n",
    "            dhash Nullable(String),\n",
    "            analysis_date DateTime64(3, 'UTC')\n",
    "        ) ENGINE = ReplacingMergeTree(analysis_date)\n",
    "        PRIMARY KEY (sha256)\n",
    "        ORDER BY (sha256)\n",
    "    \"\"\")\n",
    "\n",
    "def create_capa_dbtable(client):\n",
    "    # Create sample_capabilities table\n",
    "    client.command(\"\"\"\n",
    "        SET enable_json_type = 1;\n",
    "    \"\"\")\n",
    "    client.command(\"\"\"\n",
    "        -- Base table for CAPA analysis results\n",
    "        CREATE TABLE IF NOT EXISTS redb_capa (\n",
    "            sha256 FixedString(64),\n",
    "            md5 FixedString(32),\n",
    "            sha1 FixedString(40),\n",
    "            analysis_date DateTime64(3, 'UTC'),\n",
    "            -- Store full JSON dumps as strings for flexibility\n",
    "            capa_dump JSON CODEC(ZSTD(3)),\n",
    "            attack_dump JSON CODEC(ZSTD(3)),\n",
    "            mbc_dump JSON CODEC(ZSTD(3))\n",
    "        ) ENGINE = ReplacingMergeTree(analysis_date)\n",
    "        ORDER BY sha256;\n",
    "    \"\"\")\n",
    "\n",
    "    client.command(\"\"\"\n",
    "        -- Normalized table for capabilities and namespaces\n",
    "        CREATE TABLE IF NOT EXISTS redb_capa_capabilities (\n",
    "            sha256 FixedString(64),\n",
    "            md5 FixedString(32),\n",
    "            sha1 FixedString(40),\n",
    "            analysis_date DateTime64(3, 'UTC'),\n",
    "            namespace LowCardinality(String),\n",
    "            capability LowCardinality(String)\n",
    "        ) ENGINE = ReplacingMergeTree(analysis_date)\n",
    "        ORDER BY (sha256, namespace, capability);\n",
    "    \"\"\")\n",
    "\n",
    "    client.command(\"\"\"\n",
    "        -- Materialized view for capability statistics\n",
    "        CREATE MATERIALIZED VIEW IF NOT EXISTS redb_capa_capabilities_stats (\n",
    "            namespace LowCardinality(String),\n",
    "            capability LowCardinality(String),\n",
    "            samples_bitmap AggregateFunction(groupBitmap, UInt64),\n",
    "            occurrence_count UInt64\n",
    "        ) ENGINE = AggregatingMergeTree()\n",
    "        ORDER BY (namespace, capability)\n",
    "        AS SELECT\n",
    "            namespace,\n",
    "            capability,\n",
    "            groupBitmapState(cityHash64(sha256)) as samples_bitmap,\n",
    "            count() as occurrence_count\n",
    "        FROM redb_capa_capabilities\n",
    "        GROUP BY namespace, capability;\n",
    "    \"\"\")\n",
    "\n",
    "    client.command(\"\"\"\n",
    "        -- Normalized table for MITRE ATT&CK mappings\n",
    "        CREATE TABLE IF NOT EXISTS redb_capa_attack (\n",
    "            sha256 FixedString(64),\n",
    "            md5 FixedString(32),\n",
    "            sha1 FixedString(40),\n",
    "            analysis_date DateTime64(3, 'UTC'),\n",
    "            tactic LowCardinality(String),\n",
    "            technique LowCardinality(String),\n",
    "            technique_id LowCardinality(String)\n",
    "        ) ENGINE = ReplacingMergeTree(analysis_date)\n",
    "        ORDER BY (sha256, tactic, technique_id);\n",
    "    \"\"\")\n",
    "\n",
    "    client.command(\"\"\"\n",
    "        -- Materialized view for ATT&CK statistics\n",
    "        CREATE MATERIALIZED VIEW IF NOT EXISTS redb_capa_attack_stats (\n",
    "            tactic LowCardinality(String),\n",
    "            technique LowCardinality(String),\n",
    "            technique_id LowCardinality(String),\n",
    "            samples_bitmap AggregateFunction(groupBitmap, UInt64),\n",
    "            occurrence_count UInt64\n",
    "        ) ENGINE = AggregatingMergeTree()\n",
    "        ORDER BY (tactic, technique_id)\n",
    "        AS SELECT\n",
    "            tactic,\n",
    "            technique,\n",
    "            technique_id,\n",
    "            groupBitmapState(cityHash64(sha256)) as samples_bitmap,\n",
    "            count() as occurrence_count\n",
    "        FROM redb_capa_attack\n",
    "        GROUP BY tactic, technique, technique_id;\n",
    "    \"\"\")\n",
    "\n",
    "    client.command(\"\"\"\n",
    "        -- Normalized table for MBC mappings\n",
    "        CREATE TABLE IF NOT EXISTS redb_capa_mbc (\n",
    "            sha256 FixedString(64),\n",
    "            md5 FixedString(32),\n",
    "            sha1 FixedString(40),\n",
    "            analysis_date DateTime64(3, 'UTC'),\n",
    "            objective LowCardinality(String),\n",
    "            behavior LowCardinality(String),\n",
    "            behavior_id LowCardinality(String)\n",
    "        ) ENGINE = ReplacingMergeTree(analysis_date)\n",
    "        ORDER BY (sha256, objective, behavior_id);\n",
    "    \"\"\")\n",
    "\n",
    "    client.command(\"\"\"\n",
    "        -- Materialized view for MBC statistics\n",
    "        CREATE MATERIALIZED VIEW IF NOT EXISTS redb_capa_mbc_stats (\n",
    "            objective LowCardinality(String),\n",
    "            behavior LowCardinality(String),\n",
    "            behavior_id LowCardinality(String),\n",
    "            samples_bitmap AggregateFunction(groupBitmap, UInt64),\n",
    "            occurrence_count UInt64\n",
    "        ) ENGINE = AggregatingMergeTree()\n",
    "        ORDER BY (objective, behavior_id)\n",
    "        AS SELECT\n",
    "            objective,\n",
    "            behavior,\n",
    "            behavior_id,\n",
    "            groupBitmapState(cityHash64(sha256)) as samples_bitmap,\n",
    "            count() as occurrence_count\n",
    "        FROM redb_capa_mbc\n",
    "        GROUP BY objective, behavior, behavior_id;\n",
    "    \"\"\")\n",
    "\n",
    "def create_die_dbtable(client):\n",
    "    # Create sample_die table\n",
    "    client.command(\"\"\"\n",
    "        SET enable_json_type = 1;\n",
    "    \"\"\")\n",
    "    client.command(\"\"\"\n",
    "        CREATE TABLE IF NOT EXISTS redb_die (\n",
    "            sha256 FixedString(64),\n",
    "            md5 FixedString(32),\n",
    "            sha1 FixedString(40),\n",
    "            die_info JSON CODEC(ZSTD(3)),\n",
    "            analysis_date DateTime64(3, 'UTC')\n",
    "        ) ENGINE = ReplacingMergeTree(analysis_date)\n",
    "        PRIMARY KEY (sha256)\n",
    "        ORDER BY (sha256)\n",
    "    \"\"\")\n",
    "\n",
    "    client.command(\"\"\"\n",
    "        CREATE MATERIALIZED VIEW redb_die_mv\n",
    "        (\n",
    "            sha256 FixedString(64),\n",
    "            md5 FixedString(32),\n",
    "            sha1 FixedString(40),\n",
    "            analysis_date DateTime64(3, 'UTC'),\n",
    "            compiler LowCardinality(Nullable(String)),\n",
    "            compiler_full Nullable(String),\n",
    "            format LowCardinality(Nullable(String)),\n",
    "            format_full Nullable(String),\n",
    "            installer LowCardinality(Nullable(String)),\n",
    "            installer_full Nullable(String),\n",
    "            joiner LowCardinality(Nullable(String)),\n",
    "            joiner_full Nullable(String),\n",
    "            language LowCardinality(Nullable(String)),\n",
    "            language_full Nullable(String),\n",
    "            library LowCardinality(Nullable(String)),\n",
    "            library_full Nullable(String),\n",
    "            linker LowCardinality(Nullable(String)),\n",
    "            linker_full Nullable(String),\n",
    "            operation_system LowCardinality(Nullable(String)),\n",
    "            operation_system_full Nullable(String),\n",
    "            packer LowCardinality(Nullable(String)),\n",
    "            packer_full Nullable(String),\n",
    "            protector LowCardinality(Nullable(String)),\n",
    "            protector_full Nullable(String),\n",
    "            sfx LowCardinality(Nullable(String)),\n",
    "            sfx_full Nullable(String),\n",
    "            sign_tool LowCardinality(Nullable(String)),\n",
    "            sign_tool_full Nullable(String),\n",
    "            stub LowCardinality(Nullable(String)),\n",
    "            stub_full Nullable(String),\n",
    "            tool LowCardinality(Nullable(String)),\n",
    "            tool_full Nullable(String),   \n",
    "            -- overlay JSON CODEC(ZSTD(3)),\n",
    "            -- resource JSON CODEC(ZSTD(3)),\n",
    "            has_overlay UInt8,\n",
    "            has_resources UInt8,\n",
    "            is_packed UInt8\n",
    "        )\n",
    "        ENGINE = ReplacingMergeTree(analysis_date)\n",
    "        ORDER BY (sha256)\n",
    "        POPULATE\n",
    "        AS SELECT\n",
    "            sha256,\n",
    "            md5,\n",
    "            sha1,\n",
    "            analysis_date,\n",
    "            nullIf(JSONExtractString(toString(die_info), 'compiler'), '') AS compiler,\n",
    "            nullIf(JSONExtractString(toString(die_info), 'compiler(full)'), '') AS compiler_full,\n",
    "            nullIf(JSONExtractString(toString(die_info), 'format'), '') AS format,\n",
    "            nullIf(JSONExtractString(toString(die_info), 'format(full)'), '') AS format_full,\n",
    "            nullIf(JSONExtractString(toString(die_info), 'installer'), '') AS installer,\n",
    "            nullIf(JSONExtractString(toString(die_info), 'installer(full)'), '') AS installer_full,\n",
    "            nullIf(JSONExtractString(toString(die_info), 'joiner'), '') AS joiner,\n",
    "            nullIf(JSONExtractString(toString(die_info), 'joiner(full)'), '') AS joiner_full,\n",
    "            nullIf(JSONExtractString(toString(die_info), 'language'), '') AS language,\n",
    "            nullIf(JSONExtractString(toString(die_info), 'language(full)'), '') AS language_full,\n",
    "            nullIf(JSONExtractString(toString(die_info), 'library'), '') AS library,\n",
    "            nullIf(JSONExtractString(toString(die_info), 'library(full)'), '') AS library_full,\n",
    "            nullIf(JSONExtractString(toString(die_info), 'linker'), '') AS linker,\n",
    "            nullIf(JSONExtractString(toString(die_info), 'linker(full)'), '') AS linker_full,\n",
    "            nullIf(JSONExtractString(toString(die_info), 'operation_system'), '') AS operation_system,\n",
    "            nullIf(JSONExtractString(toString(die_info), 'operation_system(full)'), '') AS operation_system_full,\n",
    "            nullIf(JSONExtractString(toString(die_info), 'packer'), '') AS packer,\n",
    "            nullIf(JSONExtractString(toString(die_info), 'packer(full)'), '') AS packer_full,\n",
    "            nullIf(JSONExtractString(toString(die_info), 'protector'), '') AS protector,\n",
    "            nullIf(JSONExtractString(toString(die_info), 'protector(full)'), '') AS protector_full,\n",
    "            nullIf(JSONExtractString(toString(die_info), 'sfx'), '') AS sfx,\n",
    "            nullIf(JSONExtractString(toString(die_info), 'sfx(full)'), '') AS sfx_full,\n",
    "            nullIf(JSONExtractString(toString(die_info), 'sign_tool'), '') AS sign_tool,\n",
    "            nullIf(JSONExtractString(toString(die_info), 'sign_tool(full)'), '') AS sign_tool_full,\n",
    "            nullIf(JSONExtractString(toString(die_info), 'stub'), '') AS stub,\n",
    "            nullIf(JSONExtractString(toString(die_info),