![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*hvZqCVeo0VzS5F2X8lgRgA.jpeg\")
<\/figure>Introduction<\/h2>\n
Twitter is a social networking and news website where users exchange short messages known as tweets.<\/p>\n
The ease with which Twitter may be scanned is one of its biggest selling points. Hundreds of millions of intriguing individuals can be followed, and their material can be read quickly, which is convenient in today\u2019s attention-deficit society.<\/p>\n
Web scraping<\/strong> is a widely-recognized process of extracting data from a website (like Twitter). After this website data is gathered, it is then exported into a format that is more user-friendly for analysis and modeling. Python libraries such as Beautiful Soup, Selenium, and many others can assist with the web scraping process. However, one disadvantage of some of these libraries is that many require familiarity with HTML, which is not necessarily a skill you\u2019d find in the average data analyst\u2019s toolbox.<\/p>\n In order to scrape data from Twitter, in particular, the company has created something called a Twitter Developer<\/a> account. This account allows you to create and manage your own projects and apps from data scraped from the site, but it comes with some serious limitations, which are outlined in detail in the Twitter FAQ<\/a> section.<\/p>\n In this article, we will explore how to scrape Twitter data:<\/strong><\/p>\n Twint<\/a> is a Python-based advanced Twitter-scraping application that allows you to scrape Tweets from Twitter profiles without having to use Twitter\u2019s API or creating a Twitter Developer account.<\/em><\/p>\n Twint makes use of Twitter\u2019s search operators<\/a>, which allow you to scrape Tweets from specific individuals, Tweets referring to specific themes, hashtags, and trends, and sort out sensitive information like e-mail and phone numbers from Tweets.<\/p>\n Twint also creates unique Twitter queries that allow you to scrape a Twitter user\u2019s followers, Tweets they\u2019ve liked, and who they follow, all without having to utilize any login, API, Selenium, or browser emulation.<\/p>\n What kind of challenges arise when building ML models at scale<\/a>? Chris Brossman from The RealReal discusses his team\u2019s playbook for big tasks.<\/p><\/blockquote>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n Twint can be installed with the pip command or straight from git. It is advisable you install Twint through Git as it comes up with little or no issues.<\/em><\/p>\n Git Installation:<\/strong><\/p>\n Using Pip:<\/strong><\/p>\n \u2014 or \u2014<\/p>\n Pipenv:<\/strong><\/p>\n In this article we will be accessing Twint <\/strong>in two places:<\/p>\n From your command line (cmd) you will need to navigate to the twint directory:<\/p>\n From your command line, we will be searching for the key #StandStrong. We will save it as a JSON<\/em> file in a directory of the same name:<\/p>\n David Adeleke (aka \u2018Davido\u2019) announced the release of his new single \u201cSTAND STRONG\u201d FT THE SAMPLES on May 13th, 2022. Around the world, there have been some very conflicting feelings about this release, as it shows a very different aspect of Nigeria\u2019s multi-talented artists.<\/p>\n For this project, we are going to scrape data from Twitter that relates to this new song, and apply sentiment analysis.. The keyword The full code of this project can be found in my Github Repository<\/strong><\/a>. Please feel free to fork the repo and use it to build any of your future projects!<\/p>\n We will be starting this project by importing all necessary libraries that will be used when scrapping Tweets using Twint, creating sentiment analysis and visualization.<\/p>\n For the code below we are going to be scraping for the keyword The parameters used in the above code, along with their definitions, can be found here:<\/p>\n We will need to do a lot of cleaning to remove unnecessary text (and other characters) from the data before we can perform sentiment analysis on it. The practice of correcting or deleting incorrect, corrupted, improperly formatted, duplicate, or incomplete data from a dataset is known as data cleaning<\/strong>.<\/p>\n Next, a function is created to remove some characters, emojis, text, and hyperlinks that will not be needed when performing sentiment analysis on the Tweet column data:<\/p>\n\n
Twint<\/h2>\n
Twint Requirements<\/h2>\n
\n
Benefits of Using Twint<\/h2>\n
\n
Installation<\/h2>\n
git clone https:\/\/github.com\/twintproject\/twint.git<\/a>\ncd twint\npip3 install . -r requirements.txt<\/span><\/pre>\npip3 install twint<\/span><\/pre>\npip3 install \u2014 user \u2014 upgrade -e git+https:\/\/github.com\/twintproject\/twint.git@origin\/master#egg=twint<\/span><\/pre>\npipenv install -e git+https:\/\/github.com\/twintproject\/twint.git#egg=twint<\/span><\/pre>\n\n
C:\\Users\\>cd twint<\/span><\/pre>\nC:\\Users\\twint>twint -s \u201c#StandStrong\u201d -o \u201cC:\/Users\/Documents\/Python Scripts\/standstrong.json\u201d \u2014 json<\/span>C:\\Users\\twint>twint -s \u201c#StandStrong\u201d -o \u201cC:\/Users\/standstrong.csv\u201d \u2014 csv \u2014 since 2022\u201306\u201301 \/\/ Scrape Tweets and save as a csv file.<\/span><\/pre>\n![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*EH9lyBGfGg0yQrr2eSmd8g.gif\")
<\/figure>Code Walkthrough and Project Building<\/h2>\n
#StandStrong<\/code> will be our base search word.<\/p>\nimport pandas as pd\nimport numpy as np\nimport re\nimport matplotlib.pyplot as plt\nplt.style.use(\u2018fivethirtyeight\u2019)<\/span>import twint\nimport nest_asyncio\nnest_asyncio.apply() #necessary to allow asyncio to end event loop\nimport nltk\nnltk.download(\u2018punkt\u2019)\nnltk.download(\u2018wordnet\u2019)\nfrom nltk import sent_tokenize, word_tokenize\nfrom nltk.stem.snowball import SnowballStemmer\nfrom nltk.stem.wordnet import WordNetLemmatizer\nfrom nltk.corpus import stopwords\nfrom textblob import TextBlob\nfrom wordcloud import WordCloud, STOPWORDS<\/span><\/pre>\nSearch for Tweet to Scrape:<\/h2>\n
#<\/strong>StandStrong<\/code> :<\/strong><\/p>\n#Configure\ns = twint.Config()\ns.Search = \"#StandStrong\"\ns.Lang = \"en\"\n#s.Limit = 1000 # optional\ns.Since = '2022\u201305\u201301'\ns.until = '2022\u201306\u201329' <\/span>#Store script in csv file\ns.Store_csv = True\ns.Output = \"C:\/Users\/Twint_vv3.csv\"<\/span>twint.run.Search(s)<\/span><\/pre>\n\n
Lang<\/code>: Specify the language of the Tweet you want to scrape.<\/li>\nSince<\/code>: Collect Tweets that were tweeted since %Y-%m-%d %H-%M-%ss.<\/li>\nUntil<\/code>: Filter tweets up until this point in time.<\/li>\nStore_csv<\/code>: Save scraped tweet data in CSV format.<\/li>\n<\/ol>\nData Cleaning<\/h2>\n
stand_df = pd.read_csv(\"Twint_vv3.cs\")\nTweet = stand_df.filter(['tweet'])\nTweet.head(10)<\/span><\/pre>\n# create a function to clean the tweets<\/span>def cleanTxt(text):\n #Remove @mentions<\/a> and replace with blank\n text = re.sub(r'@[A-Za-z0\u20139]+', '', text)\n\n #Remove the '#' symbol, replace with blank\n text = re.sub(r'#', '', text)\n\n #Removing RT, replace with blank\n text = re.sub(r'RT[\\s]+', '', text)\n\n #Remove the hyperlinks\n text = re.sub(r'https?:\\\/\\\/\\S+', '', text)\n\n return text<\/span># clean the text\nTweet['tweet']= Tweet['tweet'].apply(cleanTxt)<\/span>#Show the clean text\nTweet<\/span><\/pre>\n![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*97zF-r6Y5xNpESuwrZlWPQ.png\")
<\/figure>
![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*a08hEfaj7Phu5F8LiUHnYA.png\")
<\/figure>![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*2FZNBNfsUWoSF-u9oZbn5w.png\")
<\/figure>![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*P2J31A5TCo0QSi-_nGk1_w.png\")
<\/figure>![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*HY_6xis1Eukz-hl3Fvlhsg.png\")
<\/figure>![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*TrcbwDfrzj8iduu_z6SQOA.png\")
<\/figure>![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*YvrrJmWCQ9xk8vDwdywtdg.png\")
<\/figure>![\"\"](\"https:\/\/miro.medium.com\/v2\/resize:fit:700\/1*bf6OzoNUm2grMNult3AwiQ.png\")
<\/figure>