Domain > deepclassiflie.org

More information on this domain is in AlienVault OTX

DNS Resolutions

Date	IP Address
2020-09-06	104.18.32.93 (ClassC)
2020-09-06	104.18.33.93 (ClassC)
2025-12-16	104.21.39.97 (ClassC)

HTTP/1.1 200 OKDate: Tue, 16 Dec 2025 08:49:41 GMTContent-Type: text/html; charsetutf-8Transfer-Encoding: chunkedConnection: keep-aliveNel: {report_to:cf-nel,success_fraction:0.0,max_age:604800}Server: cloudflareLast-Modified: Tue, 22 Aug 2023 19:14:31 GMTAccess-Control-Allow-Origin: *expires: Tue, 16 Dec 2025 08:59:41 GMTCache-Control: max-age600Report-To: {group:cf-nel,max_age:604800,endpoints:{url:https://a.nel.cloudflare.com/report/v4?sUYVcRLomK2NP7VuJ9uFGniHQ%2FT56xCZLzKqHKd475zHGRcg3PgciLwGbMevo%2B1cM9tmA4Dwe5coOqaTXOjI4C67ZvaaKfkq%2FBDPRj5pGNE4%3D}}x-proxy-cache: MISSX-GitHub-Request-Id: 0EA4:866B0:970CCA:9A002D:69411D24Age: 0Via: 1.1 varnishX-Served-By: cache-bfi-krnt7300112-BFIX-Cache: MISSX-Cache-Hits: 0X-Timer: S1765874982.878434,VS0,VE90Vary: Accept-EncodingX-Fastly-Request-ID: 67c3ed0156d00c4ff8dbae56a4e8ab053887997bcf-cache-status: DYNAMICCF-RAY: 9aecedcc9d599bf9-PDXalt-svc: h3:443; ma86400

!DOCTYPE html>html lang>  head>    title>/title>          !-- Google tag (gtag.js) -->      script async srchttps://www.googletagmanager.com/gtag/js?idG-H9RY59H33Y>/script>      script>        window.dataLayer  window.dataLayer || ;        function gtag(){dataLayer.push(arguments);}        gtag(js, new Date());                gtag(config, G-H9RY59H33Y);      /script>        meta charsetUTF-8>    meta nameviewport contentwidthdevice-width, initial-scale1 />    meta nametheme-color content#157878 />    meta nametwitter:card contentsummary_large_image />    meta nametwitter:creator content@DeepClassiflie />    meta propertytwitter:title contentDeep Classiflie />    meta nametwitter:image contenthttps://deepclassiflie.org/assets/dc_twitter_card.png />    meta propertyog:url contenthttps://deepclassiflie.org />    meta propertyog:title contentDeep Classiflie />    meta propertyog:description contentShallow fact-checking with deep neural networks />    meta propertyog:image contenthttps://deepclassiflie.org/assets/dc_twitter_card.png />    !-- Begin Jekyll SEO tag v2.8.0 -->title>Deep Classiflie | Shallow fact-checking with deep neural networks./title>meta namegenerator contentJekyll v3.9.3 />meta propertyog:title contentDeep Classiflie />meta propertyog:locale contenten_US />meta namedescription contentShallow fact-checking with deep neural networks. />meta propertyog:description contentShallow fact-checking with deep neural networks. />link relcanonical hrefhttps://deepclassiflie.org/ />meta propertyog:url contenthttps://deepclassiflie.org/ />meta propertyog:site_name contentDeep Classiflie />meta propertyog:type contentwebsite />meta nametwitter:card contentsummary />meta propertytwitter:title contentDeep Classiflie />script typeapplication/ld+json>{@context:https://schema.org,@type:WebSite,description:Shallow fact-checking with deep neural networks.,headline:Deep Classiflie,name:Deep Classiflie,url:https://deepclassiflie.org/}/script>!-- End Jekyll SEO tag -->        link relstylesheet href/assets/css/style.css?v28ac09aa9b28686e37de79bfcecef870cb8aa5ca>      /head>  body>    section classpage-header>      img classdc_logo_head src/assets/dc_logo.png altDeep Classiflie Logo />      div classcust-project-tagline>Shallow fact-checking with deep neural networks./div>      div classcust-nav-wrap>        ul>          li>a hrefindex.html classbtn>About/a>/li>          li>a hrefpred_explorer.html classbtn>Prediction Explorer/a>/li>          li>a hrefperf_explorer.html classbtn>Performance Explorer/a>/li>          li>a hrefcurrent_explorer.html classbtn>Current Predictions/a>/li>                  /ul>    /div>    /section>    section classmain-content>      details>  summary>strong>Table of Contents/strong>/summary>  ul>    li>a href#what-is-deep-classiflie>What is Deep Classiflie?/a>/li>    li>a href#project-motivation>Project Motivation/a>/li>    li>a href#model-exploration>Model Exploration/a>/li>    li>a href#core-components>Core Components/a>/li>    li>a href#current-performance>Current Performance/a>/li>    li>a href#noteworthy-features>Noteworthy Features/a>/li>    li>a href#data-pipeline>Data Pipeline/a>/li>    li>a href#configuration>Configuration/a>/li>    li>a href#further-research>Further Research/a>/li>    li>a href#model-replication>Model Replication/a>/li>    li>a href#model-replication-and-exploration-with-docker>Model Replication and Exploration w/ Docker/a>/li>    li>a href#caveats>Caveats/a>/li>    li>a href#citing-deep-classiflie>Citing Deep Classiflie/a>/li>    li>a href#references-and-notes>References and Notes/a>/li>    li>a href#license>License/a>/li>    li>a hrefhttps://github.com/speediedan/deep_classiflie>View on GitHub/a>/li>  /ul>/details>hr />h3 idwhat-is-deep-classiflie>What is Deep Classiflie?/h3>ul>  li>Deep Classiflie is a framework for developing ML models that bolster fact-checking efficiency. Predominantly a research projectsup idae>a href#ce>e/a>/sup>, I plan to extend and maintain this framework in pursuing my own research interests so am sharing it in case it’s of any utility to the broader community./li>  li>As a POC, the initial alpha release of Deep Classiflie generates/analyzes a model that continuously classifies a single individual’s statements (Donald Trump)sup ida1>a href#f1>1/a>/sup> using a single ground truth labeling source (The Washington Post)./li>  li>    p>The Deep Classiflie POC model’s a href/current_explorer>current predictions/a> and performance on the most recent test set can be a href#model-exploration>explored/a> and better understood using the a href/current_explorer>current prediction explorer/a>:/p>    p>img src/assets/current_explorer.gif altcurrent prediction explorer />/p>  /li>  li>    p>the a href/pred_explorer>prediction explorer/a>:/p>    p>img src/assets/pred_exp.gif altprediction explorer />/p>  /li>  li>    p>and the a href/perf_explorer>performance explorer/a>:/p>    p>img src/assets/temporal_confusion_matrices.gif alttemporal performance explorer />  img src/assets/conf_bucket_confusion_matrices.gif altconfidence bucket performance explorer />/p>  /li>  li>This research project initially integrated multiple fact-checkers and should be readily extensible to other statement streams/fact-checkers. The extent to which models built using this framework/approach generalize among different statement issuers is the next primary focus of this research project./li>/ul>hr />h3 idproject-motivation>Project Motivation/h3>ul>  li>    p>Asymmetries in disinformation diffusion dynamicssup ida0>a href#f0>0/a>/sup> and in rates of generation vs detection suggest building tools that maximize fact-checking efficiency could be of immense societal value. To this end, Deep Classiflie can be used to build and analyze models that leverage previous fact-checking labels to enhance future fact-checking efficiency by identifying new statements most likely to be identified as falsehoods./p>  /li>  li>With a few a href#caveats>caveats/a>, the performance of the initial POC model built using this framework is remarkably encouraging (see a href#current-performance>current performance/a> for more detail):    ul>      li>Global metrics (performance on the entire test set): br />  img src/assets/global_metrics_summ.png altGlobal Stat Summary />/li>      li>Local metrics (performance in a local region of confidence):br />  img src/assets/max_ppv_nontweets.png altMax PPV Non-Tweets /> br />  img src/assets/max_ppv_tweets.png altMax PPV Tweets />/li>    /ul>  /li>  li>Future research using this framework is intended to explore the extent to which these claim-classification models generalize across individuals and fact-checking entities. Approaches to sharing and integrating these models into a mutually-enhancing network are also to be investigated./li>/ul>hr />h3 idmodel-exploration>Model Exploration/h3>p>The best way to start understanding/exploring the current model is to use the explorers on deepclassiflie.org:/p>details>  summary>strong>a href/pred_explorer>Prediction Explorer/a>/strong>/summary>  p>Explore randomly sampled predictions from the test set of the latest model incarnation. The explorer uses a hrefhttps://captum.ai/>captum’s/a> implementation of integrated gradientssup ida7>a href#f7>7/a>/sup> to visualize attributions of statement predictions to tokens in each statement. Read more about explorer a href##current-performance>below./a>/p>  p>img src/assets/pred_exp.gif altprediction explorer />/p>/details>details>  summary>strong>a href/perf_explorer>Performance Explorer/a>/strong>/summary>  p>Explore the performance of the current model incarnation using confusion matrices oriented along temporal and confidence-based axes./p>  p>img src/assets/temporal_confusion_matrices.gif alttemporal performance explorer />img src/assets/conf_bucket_confusion_matrices.gif altconfidence bucket performance explorer />/p>/details>details>  summary>strong>a href/current_explorer>Current Predictions Explorer/a>/strong>/summary>  p>Explore current predictions of the latest model. The most recent (max 5000) statements that have yet to be labeled by the currently used fact-checking sources (only a hrefhttps://www.washingtonpost.com/graphics/politics/trump-claims-database/>Washington Post Factchecker/a> at present) are available./p>  p>Live predictions are continuously added via a hrefhttps://ipfs.io>ipfs/a>. Twitter is polled for new statements every 3 minutes, a hrefhttps://factba.se/>Factba.se/a> every 15 minutes./p>  p>This explorer provides fact-checkers a means (one of many possible) of using current model predictions and may also help those building fact-checking systems evaluate the potential utility of integrating similar models into their systems./p>  p>img src/assets/current_explorer.gif altcurrent predictions explorer />/p>/details>hr />h3 idcore-components>Core Components/h3>p>The entire initial Deep Classiflie system (raw dataset, model, analytics modules, twitter bot etc.) can be built from scratch using the publicly available code here.sup ida2>a href#f2>2/a>/sup>/p>div classabout-table>  table>    thead>      tr>        th>Component/th>        th>Description/th>      /tr>    /thead>    tbody>      tr>        td>a hrefhttps://github.com/speediedan/deep_classiflie>strong>deep_classiflie/strong>/a>/td>        td>Core framework for building, training and analyzing fact-check facilitating ML models. Can operate independently from deep_classiflie_db when training a model using existing dataset collections or when performing inference. Depends on deep_classiflie_db for certain functions such as creating new dataset collections, running the tweetbot, running the analytics modules etc. sup ida3>a href#f3>3/a>/sup>/td>      /tr>      tr>        td>a hrefhttps://github.com/speediedan/deep_classiflie_db>strong>deep_classiflie_db/strong>/a>/td>        td>Backend datastore for managing Deep Classiflie metadata, analyzing Deep Classiflie intermediate datasets and orchestrating Deep Classiflie model training pipelines. Includes data scraping modules for the initial model data sources (twitter, factba.se, washington post – politifact and the toronto star were removed from an earlier version and may be re-added among others as models for other prominent politicians are explored)/td>      /tr>    /tbody>  /table>/div>details>  summary>strong>a href#data-pipeline>Dataset Generation/a>/strong>/summary>  ul>    li>For simplicity, scrape “ground truth” falsehood labels from a single source (a hrefhttps://www.washingtonpost.com/graphics/politics/trump-claims-database>Washington Post Factchecker/a>)/li>    li>Scrape a substantial fraction of public statements from multiple sources. (a hrefhttps://factba.se>Factba.se/a>, a hrefhttps://twitter.com>Twitter/a>)/li>    li>Use statement hashes and subword representations from a base model (ALBERTsup ida8>a href#f8>8/a>/sup>)  to remove “false” statements present in the larger “truths” corpus./li>    li>Prepare chronologically disjoint train/dev/test sets (to avoid data leakage) and attempt to reduce undesired superficial class-aligned distributional artifacts that could be leveraged during model training. NNs are lazy, they’ll cheat if we let them./li>  /ul>/details>details>  summary>strong>strong>Model Training/strong>/strong>/summary>  ul>    li>Fine-tune a base model (currently HuggingFace’s a hrefhttps://huggingface.co/transformers/model_doc/albert.html>ALBERT implementation/a> with some minor customizations) in tandem with a simple embedding reflecting the semantic shift associated with the medium via which the statement was conveyed (i.e., for the POC, just learn the tweet vs non-tweet transformation) (using a hrefhttps://pytorch.org/>Pytorch/a>)/li>    li>Explore the latest model’s training session on a hrefhttps://tensorboard.dev/experiment/Ys0KLo5nRnq0soINjyEv4A>tensorboard.dev/a>./li>    li>N.B. neuro-symbolic methodssup ida6>a href#f6>6/a>/sup> that leverage knowledge bases and integrate symbolic reasoning with connectionist methods are not used in this model. Use of these approaches may be explored in a href#further-research>future research/a> using this framework./li>  /ul>/details>details>  summary>strong>strong>Analysis & Reporting/strong>/strong>/summary>  ul>    li>Interpret statement-level predictions using a hrefhttps://captum.ai/>captum’s/a> implementation of integrated gradients to visualize attributions of statement predictions to tokens in each statement./li>    li>Test set prediction and model performance exploration dashboards were built using a hrefhttps://docs.bokeh.org/en/latest/index.html>bokeh/a> and a hrefhttps://github.com/jekyll/jekyll>Jekyll/a>/li>    li>The a href/current_explorer>current prediction explorer/a> was built using a hrefhttps://datatables.net/>datatables/a> and a hrefhttps://ipfs.io>ipfs/a> with pinning provided by a hrefhttps://pinata.cloud/>pinata/a>/li>    li>Two inference daemons poll, analyze and classify new statements:      ol>        li>          p>A daemon that publishes via a hrefhttps://ipfs.io>IPFS/a> all new statement classifications and inference output./p>          p>img src/assets/current_explorer.gif altcurrent predictions explorer />/p>        /li>        li>          p>Automated false statement reports for predictions meeting the desired a hrefhttps://en.wikipedia.org/wiki/Positive_and_negative_predictive_values>PPV/a> confidence threshold can be published on twitter via a twitter bot, which leverages a hrefhttps://www.tweepy.org/>Tweepy/a>. The bot sup idah>a href#ch>h/a>/sup> tweets out a statement analysis and model interpretation “report” such as the one below for statements the model deems most likely to be labeled falsehoods (see a href#current-performance>current performance/a> for more detail):/p>          p>img src/assets/example_twitter_report.png altExample tweet report />/p>        /li>      /ol>    /li>    li>XKCD fans may notice the style of the dashboard explorers and statement reports are XKCD-inspired using the Humor Sans font created by a hrefhttps://twitter.com/ch00ftech>@ch00ftech/a>. Thanks to him (and a hrefhttps://twitter.com/xkcd>@xkcd/a> of course!)/li>  /ul>/details>hr />h3 idcurrent-performance>Current Performance/h3>details>  summary>strong>Global/strong>/summary>  p>Global metricssup ida9>a href#f9>9/a>/sup> summarized in the table below relate to the current model’s performance on a test set comprised of ~13K statements made between 2020-04-03 and 2020-07-08:br />img src/assets/global_metrics_summ.png altGlobal Stat Summary />/p>/details>details>  summary>strong>Local/strong>/summary>  p>To minimize false positives and maximize the model’s utility, the following approach is used to issue high-confidence predictions:/p>  ol>    li>All test set predictions are bucketed by model confidence (derived from the raw prediction sigmoid output)./li>    li>Various performance metrics are calculated, grouped by confidence bucket (4%/10% of test set for non-tweets/tweets respectively). Most relevantly:      ul>        li>a hrefhttps://en.wikipedia.org/wiki/Positive_and_negative_predictive_values>PPV/a>/li>        li>Positive prediction ratio: (bucket true positives + bucket false positives)/#statements in bucket/li>        li>Bucket-level accuracy/li>      /ul>    /li>    li>Report estimated local accuracy metrics of given prediction by associating it with its corresponding confidence bucket. See a href#caveats>caveats/a> regarding recognized performance biasessup idaa>a href#ca>a/a>/sup>      ul>        li>In the prediction explorer, randomly sample 100 statements (including all confusion matrix classes) from each of four confidence buckets: the maximum and minimum accuracy buckets for each statement type.br />  img src/assets/max_ppv_nontweets.png altMax PPV Non-Tweets /> br />  img src/assets/max_ppv_tweets.png altMax PPV Tweets />/li>      /ul>    /li>  /ol>/details>hr />h3 idnoteworthy-features>Noteworthy Features/h3>details>  summary>strong>Dataset generation/strong>/summary>  ul>    li>Easily and extensively configurable using yaml a href#configuration>configuration files/a>./li>    li>Multiple different class balancing strategies available (oversampling, class ratios etc.)/li>    li>“Weakly converge” superficially divergent class distributions using UnivariateDistReplicator abstraction/li>    li>Easily experiment with different train/dev/test splits/configurations via declarative DatasetCollection definitions./li>  /ul>/details>details>  summary>strong>Model training/strong>/summary>  ul>    li>Automated recursive fine-tuning of the base model with a FineTuningScheduler abstraction/li>    li>Configurable label-smoothingsup ida4>a href#f4>4/a>/sup>/li>    li>Generate and configure thawing schedules for models./li>    li>EarlyStopping easily configurable with multiple non-standard monitor metrics (e.g. mcc)/li>    li>Both automatic and manually-specified a hrefhttps://pytorch.org/blog/stochastic-weight-averaging-in-pytorch/>stochastic weight averaging/a> of model checkpointssup idaf>a href#cf>f/a>/sup>/li>    li>Mixed-precision trainingsup idag>a href#cg>g/a>/sup>/li>  /ul>/details>details>  summary>strong>Analysis & reporting/strong>/summary>  ul>    li>Extensive suite of reporting views for analyzing model performance and global and local levels/li>    li>A a href/current_explorer>current prediction explorer/a> that provides fact-checkers a means (one of many possible) of using current model predictions. This dashboard may also help those building fact-checking systems evaluate the potential utility of integrating similar models into their systems./li>    li>Statement and performance exploration dashboards for interpreting model predictions and understanding its performance/li>    li>xkcd-themed visualization of UMAP-transformed statement embeddings/li>  /ul>/details>hr />h3 iddata-pipeline>Data Pipeline/h3>p>To conserve resources and for POC research expediency, the current pipeline uses a local relational DB (MariaDB). Ultimately, a distributed data store would be preferable and warranted if this project merits sufficient interest from the community or a POC involving a distributed network of models is initiated./p>details>  summary>strong>Deep Classiflie Data Pipeline/strong>/summary>  p>img src/assets/deep_classiflie_data_pipeline.svg altDeep Classiflie Data Pipeline />/p>/details>details>  summary>strong>False Statement Filter Processes/strong>/summary>  p>img src/assets/False%20Statement%20Filter%20Processes.svg altFalse Statement Filter Processes />/p>/details>details>  summary>strong>Distribution Convergence Process/strong>/summary>  p>img src/assets/Distribution%20Convergence%20Process.svg altDistribution Convergence Process />/p>/details>details>  summary>strong>Dataset Generation Process/strong>/summary>  p>img src/assets/Dataset%20Generation%20Process.svg altDataset Generation Process />/p>/details>hr />h3 idconfiguration>Configuration/h3>p>The parameters used in all Deep Classiflie job executions related to the development of the POC model are provided in the configs directory/p>div classabout-table>  table>    thead>      tr>        th>Config File/th>        th>Description/th>      /tr>    /thead>    tbody>      tr>        td>strong>config_defaults.yaml/strong>/td>        td>default values and descriptions of all non-sql parameters/td>      /tr>      tr>        td>strong>config_defaults_sql.yaml/strong>/td>        td>default values and descriptions of all sql parameters/td>      /tr>      tr>        td>strong>dataprep_only.yaml/strong>/td>        td>parameters used to generate dataset/td>      /tr>      tr>        td>strong>train_albertbase.yaml/strong>/td>        td>parameters used to recursively train the POC model/td>      /tr>      tr>        td>strong>gen_swa_ckpt.yaml/strong>/td>        td>parameters used to generate an swa checkpoint (current release was built using swa torchcontrib module but will switch to the now-integrated pytorch swa api in the next release)/td>      /tr>      tr>        td>strong>gen_report.yaml/strong>/td>        td>parameters used to generate model analysis report(s)/td>      /tr>      tr>        td>strong>gen_dashboards.yaml/strong>/td>        td>parameters used to generate model analysis dashboards/td>      /tr>      tr>        td>strong>cust_predict.yaml/strong>/td>        td>parameters used to perform model inference on arbitrary input statements/td>      /tr>      tr>        td>strong>tweetbot.yaml/strong>/td>        td>parameters used to run the tweetbot behind @DeepClassiflie/td>      /tr>      tr>        td>strong>infsvc.yaml/strong>/td>        td>parameters used to run the inference service behind the current prediction explorer/td>      /tr>    /tbody>  /table>/div>hr />h3 idfurther-research>Further Research/h3>ul>  li>The NLP research community is actively studying precisely what these models learnsup ida5>a href#f5>5/a>/sup>. A closer examination of the distributed statement representations learned by Deep Classiflie models is planned. As of writing (2020.08.19) Google has just open-sourced an NLP model interpretability tool named a hrefhttps://github.com/pair-code/lit>LIT/a> that one could use to further elucidate the patterns identified by Deep Classiflie models. Customizing and extending LIT for exploration of Deep Classiflie model statement representations may be warranted./li>  li>The extent to which these claim-classification models generalize across individuals and fact-checking entities should be explored./li>  li>Approaches to sharing and integrating these models into a mutually-enhancing network are also to be investigated.    ul>      li>A distributed network of fact-checking model instances each built and fine-tuned for a given salient public figure using a framework such as Deep Classiflie might allow human fact-checkers to allocate their attention more efficiently to verifying high-probability falsehoods./li>      li>One can envisage a positive feedback loop developing wherein improved fact-checking coverage and latency yield increases in model-training data with the resultant improved model accuracy in turn driving further fact-checking efficiencies. The prospect of this potential value I think merits investment in research frameworks such as Deep Classiflie that attempt to explore and extend the efficacy of ML-driven enhancements to fact-checking systems./li>    /ul>  /li>/ul>hr />h3 idmodel-replication>Model Replication/h3>details>  summary>strong>Instructions/strong>/summary>  p>N.B. before you begin, the core external dependency is admin access to a mariadb or mysql DB/p>  ol>    li>Clone deep_classiflie and deep_classiflie_db (make them peer directories if you want to minimize configuration)      div classlanguage-shell highlighter-rouge>div classhighlight>pre classhighlight>code> git clone https://github.com/speediedan/deep_classiflie.git git clone https://github.com/speediedan/deep_classiflie_db.git/code>/pre>/div>      /div>    /li>    li>install a hrefhttps://docs.conda.io/projects/conda/en/latest/user-guide/install/download.html#anaconda-or-miniconda>conda/a> if necessary. Then create and activate deep_classiflie virtual env:      div classlanguage-shell highlighter-rouge>div classhighlight>pre classhighlight>code> conda span classnb>env /span>create span classnt>-f/span> ./deep_classiflie/assets/deep_classiflie.yml conda activate deep_classiflie/code>/pre>/div>      /div>    /li>    li>clone captum and HuggingFace’s transformers repos. Install transformers binaries.:      div classlanguage-shell highlighter-rouge>div classhighlight>pre classhighlight>code> git clone https://github.com/pytorch/captum.git git clone https://github.com/huggingface/transformers span classnb>cd /span>transformers pip span classnb>install/span> span classnb>./span>/code>/pre>/div>      /div>    /li>    li>      p>a hrefhttps://mariadb.com/kb/en/getting-installing-and-upgrading-mariadb/>Install mariadb/a> or mysql DB if necessary./p>    /li>    li>      p>These are the relevant DB configuration settings used for the current release of Deep Classiflie’s backend. Divergence from this configuration has not been tested and may result in unexpected behavior./p>      pre>code classlanguage-mysql> collation-server  utf8mb4_unicode_ci init-connectSET NAMES utf8mb4 character-set-server  utf8mb4 sql_mode  STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION,ANSI_QUOTES transaction-isolation  READ-COMMITTED/code>/pre>    /li>    li>copy/update relevant Deep Classiflie config file to $HOME dir      div classlanguage-shell highlighter-rouge>div classhighlight>pre classhighlight>code> span classnb>cp/span> ./deep_classiflie_db/db_setup/.dc_config.example ~ span classnb>mv/span> .dc_config.example .dc_config vi .dc_config/code>/pre>/div>      /div>      div classlanguage-shell highlighter-rouge>div classhighlight>pre classhighlight>code> span classc># configure values appropriate to your environment and move to $HOME/span> span classc># Sorry I havent had a chance to write a setup config script for this yet.../span> span classnb>export /span>span classnv>LD_LIBRARY_PATH/span>span classo>/span>span classs2>/span>span classnv>$LD_LIBRARY_PATH/span>span classs2>:/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64/span> span classnb>export /span>span classnv>CUDA_HOME/span>span classo>/span>/usr/local/cuda span classnb>export /span>span classnv>PYTHONPATH/span>span classo>/span>span classs2>/span>span classk>${/span>span classnv>PYTHONPATH/span>span classk>}/span>span classs2>:/span>span classk>${/span>span classnv>HOME/span>span classk>}/span>span classs2>/repos/edification/deep_classiflie:/span>span classk>${/span>span classnv>HOME/span>span classk>}/span>span classs2>/repos/captum:/span>span classk>${/span>span classnv>HOME/span>span classk>}/span>span classs2>/repos/transformers:/span>span classk>${/span>span classnv>HOME/span>span classk>}/span>span classs2>/repos/edification/deep_classiflie_db/span> span classnb>export /span>span classnv>DC_BASE/span>span classo>/span>span classs2>/span>span classnv>$HOME/span>span classs2>/repos/edification/deep_classiflie/span> span classnb>export /span>span classnv>DCDB_BASE/span>span classo>/span>span classs2>/span>span classnv>$HOME/span>span classs2>/repos/edification/deep_classiflie_db/span> span classnb>export /span>span classnv>PYTHONUNBUFFERED/span>span classo>/span>1 span classnb>export /span>span classnv>DCDB_PASS/span>span classo>/span>span classs2>dcbotpasshere/span> span classnb>export /span>span classnv>DCDB_USER/span>span classo>/span>span classs2>dcbot/span> span classnb>export /span>span classnv>DCDB_HOST/span>span classo>/span>span classs2>hostgoeshere/span> span classnb>export /span>span classnv>DCDB_NAME/span>span classo>/span>span classs2>deep_classiflie/span>/code>/pre>/div>      /div>    /li>    li>execute Deep Classiflie DB backend initialization script:      div classlanguage-shell highlighter-rouge>div classhighlight>pre classhighlight>code> span classnb>cd /span>deep_classiflie_db/db_setup ./deep_classiflie_db_setup.sh deep_classiflie/code>/pre>/div>      /div>      p>Ensure you have access to a DB user with administrator privs. “admin” in the case above. img srcassets/dc_schema_build.gif altDeep Classiflie logo />/p>    /li>    li>login to the backend db and seed historical tweets (necessary as only most recent 3200 can currently be retrieved directly from twitter)      pre>code classlanguage-mysql> mysql -u dcbot -p use deep_classiflie source dcbot_tweets_init_20200910.sql exit/code>/pre>    /li>    li>copy over relevant base model weights to specified model_cache_dir:      div classlanguage-shell highlighter-rouge>div classhighlight>pre classhighlight>code> span classc># model_cache_dir default found in configs/config_defaults.yaml/span> span classc># it defaults to $HOME/datasets/model_cache/deep_classiflie//span> span classnb>cd/span> span classo>{/span>PATH_TO_DEEP_CLASSIFLIE_BASEspan classo>}/span>/deep_classiflie/assets/ span classnb>cp /span>albert-base-v2-pytorch_model.bin albert-base-v2-spiece.model span classo>{/span>MODEL_CACHE_DIRspan classo>}/span>//code>/pre>/div>      /div>    /li>    li>Run deep_classiflie.py with the provided config necessary to download the raw data from the relevant data sources (factba.se, twitter, washington post), execute the data processing pipeline and generate the dataset collection.      div classlanguage-shell highlighter-rouge>div classhighlight>pre classhighlight>code>span classnb>cd /span>deep_classiflie./deep_classiflie.py span classnt>--config/span> span classs2>{PATH_TO_DEEP_CLASSIFLIE_BASE}/configs/dataprep_only.yaml/span>See relevant process diagrams to better understand the dataset generation pipeline and process./code>/pre>/div>      /div>      ul>        li>While I have set seeds for the majority of randomized processes in the data pipeline, there are a couple points in the pipeline that remain non-deterministic at the moment (see issue #). As such, the dataset generation log messages should approximate those below, but variation within 1% is expected./li>      /ul>      div classlanguage-plaintext highlighter-rouge>div classhighlight>pre classhighlight>code>(...lots of initial data download/parsing message above...) 2020-08-14 16:55:22,165:deep_classiflie:INFO: Proceeding with uninitialized base model to generate dist-based duplicate filter2020-08-14 16:55:22,501:deep_classiflie:INFO: Predictions from model weights: 2020-08-14 16:57:14,215:deep_classiflie:INFO: Generated 385220 candidates for false truth analysis2020-08-14 16:57:15,143:deep_classiflie:INFO: Deleted 7073 truths from truths table based on similarity with falsehoods enumerated in base_false_truth_del_cands2020-08-14 16:57:30,181:deep_classiflie:INFO: saved 50873 rows of a transformed truth distribution to db2020-08-14 16:57:30,192:deep_classiflie:DEBUG: DB connection obtained: <mysql.connector.pooling.PooledMySQLConnection object at 0x7f8216056e50>2020-08-14 16:57:30,220:deep_classiflie:DEBUG: DB connection closed: <mysql.connector.pooling.PooledMySQLConnection object at 0x7f8216056e50>2020-08-14 16:57:30,221:deep_classiflie:INFO: Building a balanced dataset from the following raw class data:2020-08-14 16:57:30,221:deep_classiflie:INFO: Label True: 50873 records2020-08-14 16:57:30,221:deep_classiflie:INFO: Label False: 19261 records2020-08-14 16:57:49,281:deep_classiflie:INFO: Saving features into cached file /home/speediedan/datasets/temp/deep_classiflie/train_converged_filtered.pkl2020-08-14 16:58:06,552:deep_classiflie:INFO: Saving features into cached file /home/speediedan/datasets/temp/deep_classiflie/val_converged_filtered.pkl2020-08-14 16:58:11,714:deep_classiflie:INFO: Saving features into cached file /home/speediedan/datasets/temp/deep_classiflie/test_converged_filtered.pkl2020-08-14 16:58:14,331:deep_classiflie:DEBUG: Metadata update complete, 1 record(s) affected..../code>/pre>/div>      /div>    /li>    li>Recursively train the deep classiflie POC model:      div classlanguage-shell highlighter-rouge>div classhighlight>pre classhighlight>code>span classnb>cd /span>deep_classiflie./deep_classiflie.py span classnt>--config/span> span classs2>{PATH_TO_DEEP_CLASSIFLIE_BASE}/configs/train_albertbase.yaml/span>/code>/pre>/div>      /div>    /li>    li>Generate an swa checkpoint (current release was built using swa torchcontrib module but will switch to the now-integrated pytorch swa api in the next release):      div classlanguage-shell highlighter-rouge>div classhighlight>pre classhighlight>code>span classnb>cd /span>deep_classiflie./deep_classiflie.py span classnt>--config/span> span classs2>{PATH_TO_DEEP_CLASSIFLIE_BASE}/configs/gen_swa_ckpt.yaml/span>/code>/pre>/div>      /div>    /li>    li>Generate model analysis report(s) using the generated swa checkpoint:      div classlanguage-shell highlighter-rouge>div classhighlight>pre classhighlight>code>span classc># NOTE, swa checkpoint generated in previous step must be added to gen_report.yaml/span>span classnb>cd /span>deep_classiflie./deep_classiflie.py span classnt>--config/span> span classs2>{PATH_TO_DEEP_CLASSIFLIE_BASE}/configs/gen_report.yaml/span>/code>/pre>/div>      /div>    /li>    li>Generate model analysis dashboards:      div classlanguage-shell highlighter-rouge>div classhighlight>pre classhighlight>code>span classc># NOTE, swa checkpoint generated in previous step must be added to gen_dashboards.yaml/span>span classnb>cd /span>deep_classiflie./deep_classiflie.py span classnt>--config/span> span classs2>{PATH_TO_DEEP_CLASSIFLIE_BASE}/configs/gen_dashboards.yaml/span>/code>/pre>/div>      /div>    /li>    li>      p>configure jekyll static site generator to use bokeh dashboards locally:/p>      div classlanguage-shell highlighter-rouge>div classhighlight>pre classhighlight>code>span classc>#prereqs/span>span classnb>sudo /span>apt-get span classnb>install /span>ruby-full build-essential zlib1g-devspan classc>#add ruby gems to user profile/span>span classnb>echo/span> span classs1># Install Ruby Gems to ~/gems/span> span classo>>>/span> ~/.bashrcspan classnb>echo/span> span classs1>export GEM_HOME$HOME/gems/span> span classo>>>/span> ~/.bashrcspan classnb>echo/span> span classs1>export PATH$HOME/gems/bin:$PATH/span> span classo>>>/span> ~/.bashrcspan classnb>source/span> ~/.bashrcspan classc>#install jekyll (ensure youre in the build dir (docs))/span>gem span classnb>install /span>jekyll bundlerspan classc>#to get nokogiri to install, you may need to be root/span>span classnb>sudo /span>gem span classnb>install /span>nokogirispan classc>#vi ./deep_classiflie/docs/Gemfile/span>span classnb>source/span> span classs1>https://rubygems.org/span>gem span classs1>nokogiri/span>gem span classs1>rack/span>, span classs1>~> 2.1.4/span>gem span classs1>rspec/span>gem span classs1>jekyll-theme-cayman/span>gem span classs2>github-pages/span>, span classs2>~> 207/span>, group: :jekyll_pluginsgem span classs2>activesupport/span>, span classs2>> 6.0.3.1/span>gem span classs1>jekyll-sitemap/span>gem span classs2>kramdown/span>, span classs2>>2.3.0/span>span classc>#note if just updating components, best approach is to update all /span>bundle updatespan classc>#start local server from ./deep_classiflie/docs//span>span classnb>cd/span> ./deep_classiflie/docs/bundle span classnb>exec /span>jekyll serve/code>/pre>/div>      /div>    /li>  /ol>/details>hr />h3 idmodel-replication-and-exploration-with-docker>Model Replication and Exploration with Docker/h3>details>  summary>strong>Instructions/strong>/summary>  p>br />As of writing (2020.10.11), Docker Compose does not fully support GPU provisioning so using the docker cli w/ –gpus flag here./p>  ol>    li>Pull image from docker hub      div classlanguage-shell highlighter-rouge>div classhighlight>pre classhighlight>code> span classnb>sudo /span>docker pull speediedan/deep_classiflie:v0.1.3/code>/pre>/div>      /div>    /li>    li>Recursively train model using latest dataset.      ul>        li>create a local directory to bind mount and use for exploring experiment output and start training container          div classlanguage-shell highlighter-rouge>div classhighlight>pre classhighlight>code>span classnb>mkdir/span> /tmp/docker_experiment_outputspan classnb>sudo /span>docker container run span classnt>--rm/span> span classnt>-d/span> span classnt>--gpus/span> all span classnt>--mount/span> span classnb>type/span>span classo>/span>span classnb>bind/span>,sourcespan classo>/span>/tmp/docker_experiment_output,targetspan classo>/span>/experiments span classnt>--name/span> deep_classiflie_train deep_classiflie:v0.1.3  span classse>\/span>conda run span classnt>-n/span> deep_classiflie python deep_classiflie.py span classnt>--config/span> /home/deep_classiflie/repos/deep_classiflie/configs/docker_train_albertbase.yaml /code>/pre>/div>          /div>        /li>        li>run tensorboard container to follow training progress (~6 hrs on a single GPU)          div classlanguage-plaintext highlighter-rouge>div classhighlight>pre classhighlight>code>sudo docker container run --rm -d --gpus all --mount typebind,source/tmp/docker_experiment_output,target/experiments -p 6006:6006 --workdir /experiments/deep_classiflie/logs --name deep_classiflie_tb deep_classiflie:v0.1.3 conda run -n deep_classiflie tensorboard --host 0.0.0.0 --logdir/experiments/deep_classiflie/logs --reload_multifiletrue/code>/pre>/div>          /div>        /li>      /ul>    /li>    li>Use a trained checkpoint to evaluate test performance      ul>        li>start the container with a local bind mount          div classlanguage-shell highlighter-rouge>div classhighlight>pre classhighlight>code>span classnb>sudo /span>docker container run span classnt>--rm/span> span classnt>-it/span> span classnt>--gpus/span> all span classnt>--mount/span> span classnb>type/span>span classo>/span>span classnb>bind/span>,sourcespan classo>/span>/tmp/docker_experiment_output,targetspan classo>/span>/experiments span classnt>--name/span> deep_classiflie_explore deep_classiflie:v0.1.3 /code>/pre>/div>          /div>        /li>        li>update the docker_test_only.yaml file, passing the desired inference path (e.g. /experiments/deep_classiflie/checkpoints/20201010172113/checkpoint-0.5595-29-148590.pt)          div classlanguage-shell highlighter-rouge>div classhighlight>pre classhighlight>code>vi configs/docker_test_only.yaml...inference_ckpt: span classs2>/experiments/deep_classiflie/checkpoints/20201010172113/checkpoint-0.5595-29-148590.pt/span>.../code>/pre>/div>          /div>        /li>        li>evaluate on test set          div classlanguage-shell highlighter-rouge>div classhighlight>pre classhighlight>code>conda run span classnt>-n/span> deep_classiflie python deep_classiflie.py span classnt>--config/span> /home/deep_classiflie/repos/deep_classiflie/configs/docker_test_only.yaml/code>/pre>/div>          /div>        /li>      /ul>    /li>    li>Run custom predictions      ul>        li>update model checkpoint used for predictions with the one you trained          div classlanguage-shell highlighter-rouge>div classhighlight>pre classhighlight>code>  vi /home/deep_classiflie/repos/deep_classiflie/configs/docker_cust_predict.yaml  ...  inference_ckpt: span classs2>/experiments/deep_classiflie/checkpoints/20201010172113/checkpoint-0.5595-29-148590.pt/span>  .../code>/pre>/div>          /div>        /li>        li>add tweets or statements to do inference/interpretation on as desired by modifying /home/deep_classiflie/datasets/explore_pred_interpretations.json/li>        li>generate predictions          div classlanguage-shell highlighter-rouge>div classhighlight>pre classhighlight>code>conda run span classnt>-n/span> deep_classiflie python deep_classiflie.py span classnt>--config/span> /home/deep_classiflie/repos/deep_classiflie/configs/docker_cust_predict.yaml span classnt>--pred_inputs/span> /home/deep_classiflie/datasets/explore_pred_interpretations.json/code>/pre>/div>          /div>        /li>        li>review prediction interpretation card in local host browser,          div classlanguage-shell highlighter-rouge>div classhighlight>pre classhighlight>code>chrome /tmp/docker_experiment_output/deep_classiflie/logs/20201011203013/inference_output/example_stmt_1_0.png/code>/pre>/div>          /div>        /li>      /ul>    /li>  /ol>/details>hr />h3 idcaveats>Caveats/h3>ul classfnum>    li> span classfnum idca>a/span> The distance threshold for filtering out false truths using base model embeddings matches falsehoods to their corresponding truths with high but imperfect accuracy. This fuzzy matching process will result in a modest upward performance bias in the test results. Model performance on datasets built using the noisy matching process (vs exclusively hash-based) improved by only ~2% globally with gains slightly disproportionately going to more confident buckets. This places a relatively low ceiling on the magnitude of the performance bias introduced through this filtering. The precise magnitude of this bias will be quantified in the future via one or both of the following methods a href#aa>↩/a>:/li>    ul>        li>once the projects author (or another contributor) have sufficient bandwidth to execute a comprehensive manual statement de-duplication, the results of that manual de-duplication can be compared to the noisy approach to quantify the bias./li>        li>when the next set of ground truth label data are released by the Washington Post, an estimated vs actual performance comparison can be performed/li>    /ul>    li>span classfnum idcb>b/span> The module used to bootstrap the POC models tweet history by crawling factba.se needs to be refactored and added into the initial dataset bootstrap process. This is presently one of many issues in the backlog. a href#ab>↩/a>/li>    li>span classfnum idcc>c/span> Deep Classiflie depends upon deep_classiflie_db (initially released as a separate repository) for much of its analytical and dataset generation functionality. Depending on how Deep Classiflie evolves (e.g. as it supports distributed data stores etc.), it may make more sense to integrate deep_classiflie_db back into deep_classiflie. a href#ac>↩/a>/li>    li>span classfnum idcd>d/span> Its notable that the model suffers a much higher FP ratio on tweets relative to non-tweets. Exploring tweet FPs, there are a number of plausible explanations for this discrepancy which could be explored in future research. a href#ad>↩/a>/li>    li>span classfnum idce>e/span> Still in early development, there are significant outstanding issues (e.g. no tests yet!) and code quality shortcomings galore, but any constructive thoughts or contributions are welcome. Im interested in using ML to curtail disinformation, not promulgate it, so I want to be clear --  this is essentially a fancy sentence similarity system with a lot of work put into building the dataset generation and model analysis data pipelines (I have a data engineering background, not a software engineering one).a href#ae>↩/a>/li>    li>span classfnum idcf>f/span> Previous versions used the swa module from torchcontrib before it graduated to core pytorch.a href#af>↩/a>/li>    li>span classfnum idcg>g/span> Previous versions used NVIDIAs native a hrefhttps://github.com/NVIDIA/apex>apex/a> before AMP was integrated into pytorcha href#ag>↩/a>/li>    li>span classfnum idch>h/span> N.B. This daemon may violate Twitters a hrefhttps://help.twitter.com/en/rules-and-policies/twitter-automation>policy/a> w.r.t. tweeting sensitive content if the subjects statements contain such content (no content-based filtering is included in the daemon). @DeepClassflie initially tested the Deep Classiflie twitter daemon but will post only framework-related announcements moving forward.a href#ah>↩/a>/li>/ul>hr />h3 idciting-deep-classiflie>Citing Deep Classiflie/h3>p>Please cite:/p>div classlanguage-tex highlighter-rouge>div classhighlight>pre classhighlight>code>@miscspan classp>{/span>Danspan classp>_/span>Dalespan classp>_/span>2020span classp>_/span>4046591,    author        span classp>{/span>Dan Dalespan classp>}/span>,    title         span classp>{{/span>Deep Classiflie: Shallow fact-checking with deep neural networksspan classp>}}/span>,    month         sep,    year          2020,    doi           span classp>{/span>10.5281/zenodo.4046591span classp>}/span>,    version       span classp>{/span>v0.1.3-alphaspan classp>}/span>,    publisher     span classp>{/span>Zenodospan classp>}/span>,    url           span classp>{/span>https://zenodo.org/record/4046591span classp>}/span>    span classp>}/span>/code>/pre>/div>/div>p>Feel free to star the a hrefhttps://github.com/speediedan/deep_classiflie>repo/a> as well if you find it useful or interesting. Thanks!/p>hr />h3 idreferences-and-notes>References and Notes/h3>ul classfnum>    li>span classfnum idf0>0/span> S. Vosoughi, D. Roy, S. Aral, a hrefhttp://fsnagle.org/papers/vosoughi2018spread.pdf>The spread of true and false news online./a> Science 359, 1146–1151 (2018). a href#a0>↩/a>/li>    li>span classfnum idf1>1/span> span stylefont-style:italic;> Please understand, the initial politician modeled was chosen principally to maximize the volume of freely available labeled data./span> Maximizing the probability of this POCs success meant leveraging the largest corpus of classified statements. The downside of this decision in our polarized environment unfortunately might be to allow partisan sentiment to obfuscate the core focus of this project: building tools to help curtail disinformation in our information landscape, irrespective of its source. a href#a1>↩/a>/li>    li>span classfnum idf2>2/span> Note that Twitter only makes freely accessible the most recent 3200 tweets in a users timeline. To obtain older data for the first modeled politician, one can bootstrap using the archival tweet data table provided. a href#a2>↩/a>/li>    li>span classfnum idf3>3/span> Though the entire system can be built from scratch, to make a subset of system functionality (primarily model training and inference) more easily accessible, a Docker container release is planned. a href#a3>↩/a>/li>    li>span classfnum idf4>4/span> Rafael Muller, Simon Kornblith, and Geoffrey Hinton. a hrefhttps://arxiv.org/pdf/1906.02629.pdf>When Does Label Smoothing Help?/a> NeurIPS (2019). a href#a4>↩/a>/li>    li>span classfnum idf5>5/span> Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. a hrefhttps://arxiv.org/pdf/2003.02912.pdf>A primer in BERTology: What we know about how BERT works./a> arXiv preprint arXiv:2002.12327. a href#a5>↩/a>/li>    li>span classfnum idf6>6/span> Alessandro Oltramari, Jonathan Francis, Cory Henson, Kaixin Ma, and Ruwan Wickramarachchi. 2020. a hrefhttps://arxiv.org/pdf/2003.04707.pdf>Neuro-symbolic Architectures for Context Understanding./a> arXiv preprint arXiv:2003.04707 (2020). a href#a6>↩/a>/li>    li>span classfnum idf7>7/span> Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. a hrefhttps://arxiv.org/pdf/1703.01365.pdf>Axiomatic attribution for deep networks./a> arXiv preprint arXiv:1703.01365 (2017). a href#a7>↩/a>/li>    li>span classfnum idf8>8/span> Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. a hrefhttps://arxiv.org/pdf/1909.11942>Albert: A lite bert for self-supervised learning of language representations./a> arXiv preprint arXiv:1909.11942, (2019). a href#a8>↩/a>/li>    li>span classfnum idf9>9/span> Metric definitions: a hrefhttps://en.wikipedia.org/wiki/Receiver_operating_characteristic#Area_under_the_curve>AUC/a>, a hrefhttps://en.wikipedia.org/wiki/Matthews_correlation_coefficient>MCC/a>, ACC.  simple accuracy a href#a9>↩/a>/li>/ul>hr />h3 idlicense>License/h3>p>a hrefhttps://badges.mit-license.org>img srchttps://img.shields.io/:license-mit-blue.svg?styleflat-square altLicense />/a>/p>p>span classgh_small> a hrefhttps://github.com/speediedan/deep_classiflie>View on GitHub/a> /span>/p>      footer classsite-footer>      /footer>    /section>  /body>/html>

Port 443

HTTP/1.1 200 OKDate: Tue, 16 Dec 2025 08:49:42 GMTContent-Type: text/html; charsetutf-8Transfer-Encoding: chunkedConnection: keep-aliveServer: cloudflarelast-modified: Tue, 22 Aug 2023 19:14:31 GMTaccess-control-allow-origin: *expires: Tue, 16 Dec 2025 08:59:42 GMTCache-Control: max-age600Report-To: {group:cf-nel,max_age:604800,endpoints:{url:https://a.nel.cloudflare.com/report/v4?slo4Nrzmf9lfnfG367cfw7io6P6cah5ABRhm8lMQey7%2F1h%2F%2Bt6NcvFqKclURI3gIyMCKKbsW16apO7UL3wuzw6d%2B0%2F%2BExYvEQ73AN406957Q%3D}}x-proxy-cache: MISSx-github-request-id: 662C:26EE57:951110:9803FC:69411D25Nel: {report_to:cf-nel,success_fraction:0.0,max_age:604800}Age: 0via: 1.1 varnishx-served-by: cache-sjc10080-SJCx-cache: MISSx-cache-hits: 0x-timer: S1765874982.067349,VS0,VE109vary: Accept-Encodingx-fastly-request-id: 396b96ba7b77b21e140e4e72882d4a29183d7498cf-cache-status: DYNAMICCF-RAY: 9aecedcd9c7ffef6-PDXalt-svc: h3:443; ma86400

View on OTX | View on ThreatMiner

Data with thanks to AlienVault OTX, VirusTotal, Malwr and others. [Sitemap]

Domain > deepclassiflie.org

Is this malicious?

DNS Resolutions

Port 80

Port 443