In this post, I share some useful tips and resources to prepare for the IELTS exam and how you can score 8 bands.

How to score 8 bands in IELTS - Facts, Tips and Resources

Let me start by answering your obvious question "_Why should I listen to your advice?_" Well, I've attempted my IELTS Academic exam a month ago and scored an overall of 8 bands with 8.5 in Listening and Reading, and 7.0 in Speaking and Writing. So I think I'm qualified enough to write this post that aims to help you achieve your desired score in IELTS. Of course, this post is not limited to only Academic IELTS, all the tips and resources are useful for General IELTS, too. Without further a due let's start.

I will start by stating some facts and truth bombs that some of you might not like. First and foremost, you can't go from 6.5 bands to 8 bands in a month. That's not how it works for most cases. IELTS is a language test and you can't just read some books and watch some videos and then expect that you have been proficient in English. Many people treat it like every other exam that they attempt in their school and college and think that they can mug up some material and score higher bands. Don't fall prey to those articles and videos that say "100 words for your 8 bands in writing/speaking", there is no such list.

On the other hand, many people think that they cannot achieve a good score because they are not that great in English. Truth be told, you must have a solid foundation but you don't need to know every nuance of the English language. Get your basic English grammar together and you are good to go. You must be patient, you can't learn it all in a day but keep learning.

Now coming to some tips, let me make it clear that there is not a single exhaustive list that works for everyone, so whatever I'm writing here is from my own experience and it is very much possible that some of these things may not make sense to you. I would suggest you experiment with different things and find what works for you.

---

## Tip 1 - Know your current and target bands score

It may sound very obvious but many people ignore this. I can't emphasize enough how important it is to know where you stand on the IELTS band level before starting and what band score you want to get. It is important because the preparation for different band score is different. For example, to get 6.5 bands you should study all 4 modules - Listening, Reading, Writing, and Speaking - equally as most of the time you require 6 bands in each module. But, if you want to score 7.5 or 8 bands overall, make sure that you prepare for the minimum required bands in all the modules but then focus on your strengths in which you can score more. Conventionally, Listening, and Reading are easy than Speaking, and Writing, so make sure you spend enough time on those to increase your overall band score.

## Tip 2 - Know the IELTS Examination thoroughly

The exam pattern of IELTS is very complex and it is very important to understand it thoroughly. Make sure you know all the question types of all the modules. Get familiar with both paper-based and computer-based exam format. You can read more about the pros and cons of Computer Delivered IELTS at [ieltsliz.com](https://ieltsliz.com/). Also, read how writing and speaking band scores are marked and understand how an examiner will assess your Writing and Speaking task responses on the [official IELTS website](https://www.ielts.com/results/how-ielts-is-marked/band-score-calculator).

## Tip 3 - Stop searching for the perfect material

Many test-takers spend a lot of money and time finding the perfect study material, In my opinion, it is not worth it. There are countless text as well as video material available online. Also, many test papers and mock tests are available but it is not feasible to cover every material. As we don't have a fixed syllabus from which the questions will be asked thus it is better to be realistic about the preparation. Just pick a book or find a video and start learning. Even if it improves your English by a small margin, it is worth it. At the same time, don't fall prey to the unrealistic materials that claim to cover "Alpha to Omega" of IELTS.

## Some real tips

Okay, I know you must be thinking that all these tips are superficial and do not make sense. So let me give you some _real_ tips that you can implement.

- **Learn vocabulary** - You don't need to remember fancy words but make sure you have enough words in your bag so that you can express your thoughts.
- **Practice listening to different accents** - I made this mistake that you shouldn't. Make sure you are comfortable with various accents because you will hear many different accents in the Listening test.
- **Measure your typing/writing speed by writing Essays** - If you are not fluent in writing or typing, make sure you practice it rigorously, otherwise you will not be able to complete the essay on the _D-Day_.
- **Don't waste time in generating best ideas** - Your ideas/stories in Speaking or Writing Part 2 don't matter. So make sure you don't waste much time on it. Just think about an ordinary idea and develop it.
- **Don't mix up your language in Speaking and Writing** - In Speaking you must use colloquial language whereas in Writing you must use formal language, so make sure you don't mix one with another.
---

## Resources

Here is a list of some websites, books, and videos that I used and found extremely helpful.

- **Cambridge IELTS Test Books** - No other tests come even closer to these tests. Make sure you practice all the tests from these books. Although there are a total of 15 books, the first 8 books are of the old format. There are many websites where you can download these books for free.

- [**IELTS Liz**](https://ieltsliz.com) - If you are starting with the IELTS preparation, make sure you watch each video from [Liz's youtube channel](https://www.youtube.com/user/ieltsliz). Trust me, you will not regret it.

- [**English Speaking Success Youtube Channel**](https://www.youtube.com/channel/UCiVm8XcbwS8-pcDEa5lFXIA) - The best resource for Speaking. From vocabulary to sentences, from question answers to cue cards, Keith has got you covered. Bonus points - He is damn funny.

- [**Makkar IELTS**](https://makkarielts.com/) - The only reliable list of topics for Speaking Part 2. If you haven't heard of it, what are you even doing?

- [**IELTS Up**](https://ielts-up.com) - You can practice listening and reading on this website and also check out their model answers for different writing questions. Between you and me, they also have vocabulary lists for various topics😉.

That's it for this post. Thank you for reading and I hope it will help you get your coveted score in IELTS. If you have any further doubts or questions, you can e-mail me at [jamesshah@bytetales.co](mailto:jamesshah@bytetales.co).


Let's learn Pandas DataFrames and what operations we can apply on it to manipulate data

Pandas DataFrames  - Learning Data Analysis with Python 

> ⚠️ **Note** - *This post is a part of Learning data analysis with python series. If you haven't read the first post, some of the content won't make sense. Check it out [here](https://bytetales.co/learning-data-analysis-with-python-introduction-to-pandas).*

In the previous article, we talked about Pandas Series, working with real world data and handling missing values in data. Although series are very useful but most real world datasets contain multiple rows and columns and that's why Dataframes are used much more than series. In this post, we'll talk about dataframe and some operations that we can do on dataframe objects.

- - -

## What is a DataFrame?

As we looked in the previous post, A Series is a container of scalars,A DataFrame is a container for Series. It's a dictionary like data structure for Series. A DataFrame is similar to a two-dimensional hetrogeneous tabular data(SQL table). A DataFrame is created using many different types of data such as dictionary of Series, dictionary of ndarrays/lists, a list of dictionary, etc. We'll look at some of these methods to create a DataFrame object and then we'll see some operations that we can apply on a DataFrame object to manipulate the data.

## DataFrame using dictionary of Series

```py
In[1]:
    d = {
    'col1' : pd.Series([1,2,3], index = ["row1", "row2", "row3"]),
    'col2' : pd.Series([4,5,6], index = ["row1", "row2", "row3"])
    }

    df = pd.DataFrame(d)

Out[1]:
        col1	col2
row1	 1	     4
row2	 2	     5
row3	 3	     6
```

As shown in above code, the keys of dict of Series becomes column names of the DataFrame and the index of the Series becomes the row name and all the data gets mapped by the row name i.e.,order of the index in the Series doesn't matter.

## DataFrame using ndarrays/lists

```py
In[2]:
    d = {
        'one' : [1.,2.,3.],
        'two' : [4.,5.,6.]
    }

    df = pd.DataFrame(d)

Out[2]:
    one	 two
0	1.0	 4.0
1	2.0	 5.0
2	3.0	 6.0
```

As shown in the above code, when we use ndarrays/lists, if we don't pass the index then the `range(n)` becomes the index of the DataFrame.And while using the ndarray to create a DataFrame, the length of these arrays must be same and if we pass an explicit index then the length of this index must also be of same length as the length of the arrays.

## DataFrame using list of dictionaries

```py
In[3]:
    d = [
        {'one': 1, 'two' : 2, 'three': 3},
        {'one': 10, 'two': 20, 'three': 30, 'four': 40}
    ]

    df = pd.DataFrame(d)

Out[3]:
    one	two	three four
0	 1	 2	 3	   NaN
1	 10	 20	 30	   40.0


In[4]:
    df = pd.DataFrame(d, index=["first", "second"])

Out[4]:

        one	two	three four
first	 1	 2	 3 	   NaN
second	 10	 20	 30	   40.0
```

And finally, as described above we can create a DataFrame object using a list of dictionary and we can provide an explicit index in this method,too.

Although learning to create a DataFrame object using these methods is necessary but in real world, we won't be using these methods to create a DataFrame but we'll be using external data files to load data and manipulate that data. So, let's take a look how to load a csv file and create a DataFrame.In the previous post, we worked with the Nifty50 data to demonstrate how Series works and similarly in this post, we'll load Nifty50 2018 data, but in this dataset we have data of Open, Close, High and Low value of Nifty50. First let's see what this dataset looks like and then we'll load it into a DataFrame.

![Nifty50 Data](https://res.cloudinary.com/bytetales/image/upload/v1599051299/nifty_50.png)

```python
In[5]:
    df = pd.read_csv('NIFTY50_2018.csv')

Out[5]:
        Date	  Open	      High	      Low	     Close
0	31 Dec 2018	 10913.20	10923.55	10853.20	10862.55
1	28 Dec 2018	 10820.95	10893.60	10817.15	10859.90
2	27 Dec 2018	 10817.90	10834.20	10764.45	10779.80
3	26 Dec 2018	 10635.45	10747.50	10534.55	10729.85
4	24 Dec 2018	 10780.90	10782.30	10649.25	10663.50
...		...		    ...        ...         ...	       ...
241	05 Jan 2018	 10534.25	10566.10	10520.10	10558.85
242	04 Jan 2018	 10469.40	10513.00	10441.45	10504.80
243	03 Jan 2018	 10482.65	10503.60	10429.55	10443.20
244	02 Jan 2018	 10477.55	10495.20	10404.65	10442.20
245	01 Jan 2018	 10531.70	10537.85	10423.10	10435.55

In[6]:
    df = pd.read_csv('NIFTY50_2018.csv', index_col=0)

Out[6]:
                  Open	      High	      Low	     Close
    Date
31 Dec 2018	    10913.20	10923.55	10853.20	10862.55
28 Dec 2018	    10820.95	10893.60	10817.15	10859.90
27 Dec 2018	    10817.90	10834.20	10764.45	10779.80
26 Dec 2018	    10635.45	10747.50	10534.55	10729.85
24 Dec 2018	    10780.90	10782.30	10649.25	10663.50
     ...		   ...        ...         ...          ...
05 Jan 2018	    10534.25	10566.10	10520.10	10558.85
04 Jan 2018	    10469.40	10513.00	10441.45	10504.80
03 Jan 2018	    10482.65	10503.60	10429.55	10443.20
02 Jan 2018	    10477.55	10495.20	10404.65	10442.20
01 Jan 2018	    10531.70	10537.85	10423.10	10435.55
```

As shown above, we have loaded the dataset and created a DataFrame called df and looking at the data, we can see that we can set the index of our DataFrame to the `Date` column and in the second cell we did that by providing the `index_col` parameter in the `read_csv` method.

There are many more parameters available in the `read_csv` method such as `usecols` using which we can deliberately ask the pandas to only load provided columns, `na_values` to provide explicit values that pandas should identify as null values and so on and so forth. Read more about all the parameters in pandas [documentation](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html).

Now, let's look at some of the basic operations that we can perform on the dataframe object in order to learn more about our data.

```python
In[7]:
    # Shape(Number of rows and columns) of the DataFrame
    df.shape

Out[7]:
    (246,4)

In[8]:
    # List of index
    df.index

Out[8]:
    Index(['31 Dec 2018', '28 Dec 2018', '27 Dec 2018', '26 Dec 2018',
       '24 Dec 2018', '21 Dec 2018', '20 Dec 2018', '19 Dec 2018',
       '18 Dec 2018', '17 Dec 2018',
       ...
       '12 Jan 2018', '11 Jan 2018', '10 Jan 2018', '09 Jan 2018',
       '08 Jan 2018', '05 Jan 2018', '04 Jan 2018', '03 Jan 2018',
       '02 Jan 2018', '01 Jan 2018'],
      dtype='object', name='Date', length=246)

In[9]:
    # List of columns
    df.columns

Out[9]:
    Index(['Open', 'High', 'Low', 'Close'], dtype='object')

In[10]:
    # Check if a DataFrame is empty or not
    df.empty

Out[10]:
    False
```

It's very crucial to know data types of all the columns because sometimes due to corrupt data or missing data, pandas may identify numeric data as 'object' data-type which isn't desired as numeric operations on the 'object' type of data is costlier in terms of time than on `float64` or `int64` i.e numeric datatypes.

```python
In[11]:
    # Datatypes of all the columns
    df.dtypes

Out[11]:
    Open     float64
    High     float64
    Low      float64
    Close    float64
    dtype: object
```

We can use iloc and loc to index and get the particular data from our dataframe.

```python
In[12]:
    # Indexing using implicit index
    df.iloc[0]

Out[12]:
    Open     10913.20
    High     10923.55
    Low      10853.20
    Close    10862.55
    Name: 31 Dec 2018, dtype: float64

In[13]:
    # Indexing using explicit index
    df.loc["01 Jan 2018"]

Out[13]:
    Open     10531.70
    High     10537.85
    Low      10423.10
    Close    10435.55
    Name: 01 Jan 2018, dtype: float64
```

We can also use both row and column to index and get specific cell from our dataframe.

```python
In[14]:
    # Indexing using both the axes(rows and columns)
    df.loc["01 Jan 2018", "High"]

Out[14]:
    10537.85
```

We can also perform all the math operations on a dataframe object same as we did on series.

```python
In[15]:
    # Basic math operations
    df.add(10)

Out[15]:
                      Open	      High	      Low	     Close
    Date
31 Dec 2018	    10923.20	10933.55	10863.20	10872.55
28 Dec 2018	    10830.95	10903.60	10827.15	10869.90
27 Dec 2018	    10827.90	10844.20	10774.45	10789.80
26 Dec 2018	    10645.45	10757.50	10544.55	10739.85
24 Dec 2018	    10790.90	10792.30	10659.25	10673.50
     ...		   ...        ...         ...          ...
05 Jan 2018	    10544.25	10576.10	10530.10	10568.85
04 Jan 2018	    10479.40	10523.00	10451.45	10514.80
03 Jan 2018	    10492.65	10513.60	10439.55	10453.20
02 Jan 2018	    10487.55	10505.20	10414.65	10452.20
01 Jan 2018	    10541.70	10547.85	10433.10	10445.55
```

We can also aggregate the data using the `agg` method. For instance, we can get the mean and median values from all the columns in our data using this method as show below.

```python
In[16]:
    # Aggregate one or more operations
    df.agg(["mean", "median"])

Out[16]:
            Open	        High	        Low	           Close
mean	10758.260366	10801.753252	10695.351423	10749.392276
median	10704.100000	10749.850000	10638.100000	10693.000000
```

However, pandas provide a more convenient method to get a lot more than just minimum and maximum values across all the columns in our data. And that method is `describe`. As the name suggests, it describes our dataframe by applying mathematical and statistical operations across all the columns.

```python
In[17]:
    df.describe()

Out[17]:
            Open	       High	           Low	           Close
count	246.000000	    246.000000	    246.000000	    246.000000
mean	10758.260366	10801.753252	10695.351423	10749.392276
std	    388.216617	    379.159873	    387.680138	    382.632569
min	    9968.800000	    10027.700000	9951.900000	    9998.050000
25%	    10515.125000	10558.650000	10442.687500	10498.912500
50%	    10704.100000	10749.850000	10638.100000	10693.000000
75%	    10943.100000	10988.075000	10878.262500	10950.850000
max	    11751.800000	11760.200000	11710.500000	11738.500000
```

And to get the name, data types and number of non-null values in each columns, pandas provide `info` method.

```python
In[18]:
    df.info()

Out[18]:
    <class 'pandas.core.frame.DataFrame'>
    Index: 246 entries, 31 Dec 2018 to 01 Jan 2018
    Data columns (total 4 columns):
    #   Column  Non-Null Count  Dtype
    ---  ------  --------------  -----
    0   Open    246 non-null    float64
    1   High    246 non-null    float64
    2   Low     246 non-null    float64
    3   Close   246 non-null    float64
    dtypes: float64(4)
    memory usage: 19.6+ KB
```

We are working with a small data with less than 300 rows and thus, we can work with all the rows but when we have tens or hundreds of thousand rows in our data, it's very difficult to work with such huge number of data. In statistics, 'sampling' is a technique that solves this problem. Sampling means to choose a small amount of data from the whole dataset such that the sampling dataset contains somewhat similar features in terms of diversity as that of the whole dataset. Now, it's almost impossible to manually select such peculiar rows but as always, pandas comes to our rescue with the `sample` method.

```python
In[19]:
    # Data Sampling -  Get random n examples from the data.
    df.sample(5)

Out[19]:
              Open	      High	      Low	      Close
Date
04 Jul 2018	10715.00	10777.15	10677.75	10769.90
22 Jun 2018	10742.70	10837.00	10710.45	10821.85
14 Mar 2018	10393.05	10420.35	10336.30	10410.90
09 Jan 2018	10645.10	10659.15	10603.60	10637.00
27 Apr 2018	10651.65	10719.80	10647.55	10692.30
```

But, executing this method produces different results everytime and that may be unacceptable in some cases. But that can be solved by providing `random_state` parameter in the `sample` method to reproduce same result everytime.

- - -

As shown above, we can perform many operations on the DataFrame object to get information of the DataFrame and from the DataFrame. These are just basic operations that we can perform on the DataFrame object, there are many more interesting methods and operations that we can perform on the DataFrame object such as `pivot` , `merge` , `join` and many more. Also, in this given dataset, we have time as the index of our DataFrame i.e this is the TimeSeries dataset and pandas also provide many methods to manipulate the TimeSeries data such as `rolling_window`.

That will be all for this post. In the next post we'll look at some of these methods and we'll perform 5 analysis tasks using these methods. Till then, you can take a look at the [pandas documentation](https://pandas.pydata.org/pandas-docs/stable/index.html) and find more information about DataFrame objects and the methods that can be applied on the DataFrame object.

**Thank you for reading**

Learn how to load and work with real world dataset with pandas and how to handle missing values in the data.

Learning Data Analysis With Python - Missing Values and Real World Data

> ⚠️  _This post is a part of Learning data analysis with python series. If you haven't read the first post, some of the content won't make sense. Check it out [here](https://bytetales.co/learning-data-analysis-with-python-introduction-to-pandas)._

In the previous article, we talked about [pandas](https://pandas.pydata.org/) library in python, Series data structure, different ways to create a Series object using a list,a dictionary and a numpy array. and how we can apply some functions on it to get some statistical data from it. In this post, we'll talk about how you can create a Series object from an external file(csv), what is loc and iloc and some tasks that we can achieve using Series. Also, we'll see an introduction to the DataFrame object as in real world data, almost in every case we come across tabular data(data with multiple columns and rows) rather than a single column data and thus, we'll be using DataFrame more frequently than Series.

## What is loc and iloc?

**loc** and **iloc** are probably the most asked topic in the data science interviews and many newcomers and data science aspirants get confused initially about loc and iloc. However, it's not rocket science and if you understand loc and iloc in Series objects, it will be easier for you to understand how to use them for indexing the DataFrame object.

In the last post, we saw that we can give index of our choice for the convenience of accessing data using that index but sometimes we may want to use the default integer indexing that pandas provide (0,1,2,...) to access some data. For example, let's say, we have data consisting 100 rows and you want to get the first 50 rows, we can do this using slicing but now that we have custom indexing, we must know the first and 50th index to get the data. This is just an oversimplified case of such scenarios, as in real world data analysis tasks you may be working on a dataset with thousands or millions of rows. In that case, it is not feasible to find custom index and then access data.
And that's when loc and iloc come-in handy. loc and iloc are builtin pandas method for indexing data from a pandas Series or DataFrame. In plain english `loc` is label based indexing which means that use name(custom index) of row and column whereas `iloc` is integer index-based which means we use integer index(default pandas index) of row and column to access the data. Let's look at the code and it will make things more clear.

```py
In[1]:
    s = pd.Series([0.0, 1, 1, 2, 3, 5, 8], index=[1, 2, 3, 4, 5, 6, 7])

Out[1]:
    1    0.0
    2    1.0
    3    1.0
    4    2.0
    5    3.0
    6    5.0
    7    8.0
    dtype: float64

In[2]:
    s.loc[3] 

Out[2]:
    1.0

In[3]:
    s.iloc[3] 

Out[3]:
    2.0
```

As you can see, in our Series object `s`, we have given custom index starting from 1 to 7. And then, when we use the `loc` method on this Series and pass 3 as index, we get the data corresponding to the 3 in our index column. And when we use the `iloc` method on this Series and again pass 3 as index, we get the data corresponding to the 3 in the default pandas index and as the dafault pandas indexing starts from 0, we get the data from 4th row. We can use the `reset_index` method to reset the index and get the default interger indexing in our Series. Looking at both the indexing side by side will make things more clear.

```
In[4]:
    s.reset_index()

Out[4]:

        index	0
    0	  1	   0.0
    1	  2	   1.0
    2	  3	   1.0
    3	  4	   2.0
    4	  5	   3.0
    5	  6	   5.0
    6	  7	   8.0
```

As you can see, the first unnamed column represents the default pandas indexing, second column named as 'index' represents the custom index that we provided and the third column named as '0' represents corresponding values. I hope now you get the idea of what is `loc` and `iloc` and what's the difference between them. We'll talk more about it in the DataFrame objects article.

___

## Missing values or NaNs

Missing values or NaN(Not A Number) values are very common in real world datasets and before using such datasets we have to handle these missing values and NaNs in our dataset in the data cleaning phase. There are many different ways in which we can handle missing values such as dropping rows that contain missing value or replacing missing values with some other values such as mean of that column or 0 or any other value. It's a subjective matter that how you handle the missing values but anyways, Pandas provide very easy way to find missing values using `isnan(), isna(), notna()` in our data and using it's advance indexing methods, changing those missing values become very easy. Now let's look at the very basic example of handling missing values by changing it to the mean of that column.

```py

In[5]:
    '''Series of Mass of Planets, where index is the name of the planet and the corresponding value is the mass of that planet.'''
    mass = pd.Series([0.33, 4.87, 5.97, 0.642, 1898, 568, 86.8, 102, 0.0146],
                    index=['Mercury', 'Venus', 'Earth', 'Mars', 'Jupiter', 'Saturn', 'Uranus', 'Neptune', 'Pluto'])
    print(mass)

Out[5]:
    Mercury       0.3300
    Venus         4.8700
    Earth         5.9700
    Mars          0.6420
    Jupiter    1898.0000
    Saturn      568.0000
    Uranus       86.8000
    Neptune     102.0000
    Pluto         0.0146
    dtype: float64

In[6]:
    '''Series of Diameter of Planets, where index is the name of the planet and the corresponding value is the diameter of that planet.'''
    diameter = pd.Series([4879, 12104, 12756, 3475, 6792, 142984, 120536, 51118, 49528, 2370],
                        index=['Mercury', 'Venus', 'Earth', 'Moon', 'Mars', 'Jupiter', 'Saturn',  'Uranus', 'Neptune', 'Pluto'])

    print(diameter)

Out[6]:
    Mercury      4879
    Venus       12104
    Earth       12756
    Moon         3475
    Mars         6792
    Jupiter    142984
    Saturn     120536
    Uranus      51118
    Neptune     49528
    Pluto        2370
    dtype: int64

In[7]:
    # Finding density using the mass and diameter series.
    density = mass / (np.pi * np.power(diameter, 3) / 6)
    print(density)

Out[7]:
    Earth      5.493286e-12
    Jupiter    1.240039e-12
    Mars       3.913302e-12
    Mercury    5.426538e-12
    Moon                NaN
    Neptune    1.603427e-12
    Pluto      2.094639e-12
    Saturn     6.194402e-13
    Uranus     1.241079e-12
    Venus      5.244977e-12
    dtype: float64

In[8]:
    pd.isna(density)

Out[8]:
    Earth      False
    Jupiter    False
    Mars       False
    Mercury    False
    Moon        True
    Neptune    False
    Pluto      False
    Saturn     False
    Uranus     False
    Venus      False
    dtype: bool

In[9]:
    # Get the index of data in density where the value is NaN and change it to the mean of density.
    density[pd.isna(density)] = np.mean(density)
    print(density)

Out[9]:
    Earth      5.493286e-12
    Jupiter    1.240039e-12
    Mars       3.913302e-12
    Mercury    5.426538e-12
    Moon       2.986303e-12
    Neptune    1.603427e-12
    Pluto      2.094639e-12
    Saturn     6.194402e-13
    Uranus     1.241079e-12
    Venus      5.244977e-12
    dtype: float64
```

If that was a lot to take in, let me walk you through it. First I've created two Series objects named mass and diameter containing mass and diameter of the planets respectively. The only change in these two is that diameter Series contains 'Moon' in it but mass doesn't. And thus, when we find density using these two Series we get the NaN value for Moon as mass doesn't contain it. I infer you thinking _"How did this happen?"_. Let me explain, when we merge two Series or create a new Series by applying some functions on two different Series, what pandas does is that it matches both the Series by their indices. Also, the order of the indices doesn't matter. Thus, when we created a new Series 'density' using 'mass' and 'diameter', pandas did the calculations and merged these two Series by their indices but in the case 'Moon' it couldn't find the 'mass' and thus, it added NaN.

After creating this new 'density' Series, we see that we have NaN value in our data and to change it we have to find it's index. Now, in our example for the sake of simplicity, we had a single NaN value and also our Series had only 10 rows so we could have changed the NaN value manually using the index of 'Moon', but with large data and many more NaN values we can't go through all the rows and find the indices of each NaN values, therefore pandas has a builtin function to find all the values in which has NaN values. And using the indexing techniques that pandas offer we changed the value of Moon's diameter in one line, otherwise we would have to go through all the data in the Series using a for loop and change the NaN values with the mean value.

As you can see, pandas package provides builtin methods for all the major tasks that we have to do while working with data and it uses numpy library that makes these operations fast as numpy use vectorization methods for all the computer heavy tasks and that's the reason why [numpy](https://numpy.org/) and [pandas](https://pandas.pydata.org/) are the most popular python packages for data science.

___

## Reading Data From External File & Working with Real World Data

Till now, we have created and worked with the data that we created using list and dictionaries but we don't create data by ourself in real world tasks, we use the data from external files such as csv(comma seperated values) files, text files, excels files, etc. And pandas provides many builtin methods to load the data from external files and make Series and DataFrame objects from that data very easily. Let's look at the example of loading 'csv' file and creating a Series object from it.

We have a real world data of closing value of Nifty index from the 1st January 2019 to 31st December 2019. [Nifty](https://en.wikipedia.org/wiki/NIFTY_50) is a benchmark Indian stock market index that represents the weighted average of 50 of the largest Indian companies listed on NSE. Let's see what this data looks like.

![Nifty Data](https://res.cloudinary.com/bytetales/image/upload/v1594893284/nifty_data.png)  

Now let's see how to load this data in our code and create a Series object. Pandas has a method named `read_csv()` which takes file path and some other parameters such as`seperator,skip_rows,header, index_col` and many [more](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html) and creates a DataFrame object from it. But we want to create a Series object, thus we'll use `iloc` and get only the first column from the DataFrame that padas creates.

```py

In[10]:
    # Using first column as index column create a Series from 'nifty_data.csv
    nifty = pd.read_csv("nifty_data.csv", index_col = 0).iloc[:,0]
    print(nifty.head(5)) # Print first 5 rows

Out[10]:
    Date
    01-Jan-2019    10910.10
    02-Jan-2019    10792.50
    03-Jan-2019    10672.25
    04-Jan-2019    10727.35
    07-Jan-2019    10771.80

```

Now, let's look at some basic operations on this data such as mean, median, max and other statistical data. But instead of doing all these operations one by on the Series, we can use a builtin pandas method `describe()` to get all these data together using a single function.

```py

In[11]:
    nifty.describe()

Out[11]:
    count      245.000000
    mean     11432.632245
    std        454.214611
    min      10604.350000
    25%      11003.050000
    50%      11512.400000
    75%      11841.550000
    max      12271.800000
    Name: Close, dtype: float64
```

And just like that, we can easily get all the statistics information from the data with a single line. Here 25%,50% and 75% represents percentile of the data meaning that 25% of values in our data is below 11003.050000 and so on. You may ask that here we dont' have median of the data but that's because the median is equal to 50th percentile ,thus 50% also represents median of the data. From a little statistics knowledge I can infer that there are not many significant outliers as there isn't much difference between mean and median.

Now remember I told you, when we apply some operations on two Series and get a new Series, pandas match them using their indices. Let's see that with another example and why keeping that mind is important. Let's say you want to find the total days on which the closing value was higher than the previous day's closing value. Well that's pretty simple, we can just take two slices from this Series, first being the data without the first column and second being the data without the last column and we can just subtract and see if the result is greater than zero or not.Right? Let's see what happens when we do that.

```py
In[12]:
    nifty[1:]

Out[12]:
    Date
    02-Jan-2019    10792.50
    03-Jan-2019    10672.25
    04-Jan-2019    10727.35
    07-Jan-2019    10771.80
    08-Jan-2019    10802.15
        ...          ...
    24-Dec-2019    12214.55
    26-Dec-2019    12126.55
    27-Dec-2019    12245.80
    30-Dec-2019    12255.85
    31-Dec-2019    12168.45
    Name: Close, Length: 244, dtype: float64

In[13]:
    nifty[:-1]

Out[13]:
    Date
    01-Jan-2019    10910.10
    02-Jan-2019    10792.50
    03-Jan-2019    10672.25
    04-Jan-2019    10727.35
    07-Jan-2019    10771.80
        ...          ...
    23-Dec-2019    12262.75
    24-Dec-2019    12214.55
    26-Dec-2019    12126.55
    27-Dec-2019    12245.80
    30-Dec-2019    12255.85
    Name: Close, Length: 244, dtype: float64

In[14]:
    nifty[1:] - nifty[:-1]

Out[14]:
    Date
    01-Apr-2019    0.0
    01-Aug-2019    0.0
    01-Feb-2019    0.0
    01-Jan-2019    NaN
    01-Jul-2019    0.0
                ...
    31-Dec-2019    NaN
    31-Jan-2019    0.0
    31-Jul-2019    0.0
    31-May-2019    0.0
    31-Oct-2019    0.0
    Name: Close, Length: 245, dtype: float64

```

Ooops!! That's not what we expected it to be right? Yes and that's because pandas matched indices of both the slices and subtracted their corresponding values and for the values that it couldn't find matching indices (01-Jan-2019 and 31-Dec-2019), it addded NaN values. Thus, when we want to subtract two series by element by element and don't want the index to be matched, we can use `values()` method on these slices and get the numpy array of values for both the slices and then subtract them.

```py
In[15]:
    (nifty.values[1:] - nifty.values[:-1]) > 0

Out[15]:
    array([False, False,  True,  True,  True,  True, False, False, False,........,False])

In[16]:
    np.sum((nifty.values[1:] - nifty.values[:-1]) > 0)

Out[16]:
    129
```

Well that's pretty much it for this post and Series object. There are many more things that you can do with the pandas Series object but including each and everything in my posts is pretty much impossible. Look at the pandas [documentation](https://pandas.pydata.org/pandas-docs/stable/index.html) and see what you can do with the Series object. In the next post, We'll talk about DataFrame objects and all the good things we can do with it. **Thank you for reading.**

---

Get the full code notebook here : [Github](https://github.com/jamesshah/pandas-notebooks/blob/master/pandas-2.ipynb)

Get the Nifty 50 data here : [Data](https://github.com/jamesshah/pandas-notebooks/blob/master/nifty_data.csv)


Let's learn Pandas - the most useful python package for data analysis and data science

Learning Data Analysis with Python - Introduction to Pandas


## What is Pandas and Why do we need it

[Pandas](https://pandas.pydata.org/) is python library that provides rich data structures and functions that makes working with relational and structured data easy, fast and convenient. It is developed on top of the [Numpy](https://numpy.org/) package for the high performance computing and it gives flexible data manipulation techniques of relational databases. Okay, now why do we need panadas at all? If we want to mainpulate high dimensional data, we have numpy, so why do we need it?
Well, indeed numpy is there for the high dimensional data manipulation but what it doesn't provide is a way to index or label our data like this.

![Pandas DataFrame](https://res.cloudinary.com/bytetales/image/upload/q_auto/w_500/v1593948383/pandas_df.webp)

And, although we can do so many interesting things to our data using numpy functions such as finding statistics like mean,median,mode and standard deviation but we can't fill missing values in our data. Numpy doesn't provide any built-in method to handle missing values in our data and in real world problems, we don't always come around the perfect datasets.

There are other limitations to numpy such as no way to group data and no way to pivot data, and thus, Pandas was created to help programmers and data scientists with the data analysis task in Python.

## Data Structures in Pandas

Well as we've mentioned earlier, pandas provides rich data structures and functions, let's start by looking at the data structures and then we'll look at the functions and methods that we can apply on these data structures to crunch data.

Pandas provide two major data structures namely "Series" and "DataFrames". In plain english, Series is a one-dimensional labeled array, whereas DataFrames is two-dimensional labeled array with columns of potentially different types. Although, Series are useful in many tasks, DataFrames are the most commonly used pandas object. You can think of DataFrame as an SQL table.

Let's look at the example for both of these data structures.

![Series & DataFrames](https://res.cloudinary.com/bytetales/image/upload/q_auto/w_500/v1593947634/series-and-dataframe.width-1200_tcd073.webp)

Let's start by looking at the different ways of creating a Series object.But, Before we get into coding, we need to install pandas as python package. This can be done using `pip install pandas` or you can install it using `conda install pandas` if you are using Anaconda Distribution for Python. If you don't have python installed on your machine, the best way to get started without all the hustle of installing it is Google Colab. Learn more about it [here](https://colab.research.google.com/notebooks/intro.ipynb). I'm going to use jupyter notebooks which you can find [here](https://github.com/jamesshah/pandas-notebooks/blob/master/pandas-1.ipynb).

## Series

As mentioned above, Series is a one-dimensional array-like object containing an array of data and an array of labels associated with the data. The data labels are called 'index'.
We can create a Series from a list or a dictionary or a numpy array.
Let's start with a list.

## Series using a list

```py
In[1]:
    import pandas as pd
    import numpy as np

    friends = ["Monica", "Chandler", "Joey", "Pheobe", "Ross", "Rachel"]
    series = pd.Series(friends)
    series

Out[1]:
    0      Monica
    1    Chandler
    2        Joey
    3      Pheobe
    4        Ross
    5      Rachel
    dtype: object
```

As we can see, we get the Series object with the data that we provided with the associated index of for each row. We can access our data from this `series` using the index just like we do with the list.

```py
In[2]:
    series[0]

Out[2]:
    'Monica'
```

Or we can get all the values in a list using `.value` attribute on series object.

```py
In[3]:
    series.values

Out[3]:
    array(['Monica', 'Chandler', 'Joey', 'Pheobe', 'Ross', 'Rachel'],
      dtype=object)
```

Similarly, we can get all the indices in a list using `.index` attribute.

```py
In[4]:
    series.index

Out[4]:
    RangeIndex(start=0, stop=6, step=1)
```

Note something different? Right, it doesn't return a list of indices as it did in values, that's because the implicit index that pandas automatically creates for us is just a list of consecutive numbers starting from 0. A more intiuative approach would be to provide index identifying each data point while creating a Series.

```py
In[5]:
    nick_names = ["Monana", "Officer Bing", "Ken Adams", "Regina Filange", "Dinosaur Guy", "Fun Aunt Rachel"]
    series = pd.Series(data = friends, index = nick_names)
    series

Out[6]:
    Monana               Monica
    Officer Bing       Chandler
    Ken Adams              Joey
    Regina Filange       Pheobe
    Dinosaur Guy           Ross
    Fun Aunt Rachel      Rachel
    dtype: object
```

And now, we can access our data using the index that we provided like this.

```py
In[7]:
    series["Dinosaur Guy"]

Out[7]:
    'Ross'
```

**_It's clearly a bad example of indexing but you got the idea, right?_**

## Series using a dictionary

Now, let's instantiate Series from a dictionary. When creating a Series from dictionaries, we don't pass index, the Series gets ordered by dictionary's insertion order and the keys of dictionary becomes indices of Series.

```py
In[8]:
    d = {"a" : 1, "b" : 2, "c" : 3}
    series = pd.Series(d)
    series

Out[8]:
    a    1
    b    2
    c    3
    dtype: int64
```

Series object also works like dictionaries. You can get or set the values in the Series by index label just like dictionaries.

```py
In[9]:
    series["c"]

Out[9]:
    3

In[10]:
    series["d"] = 4
    series

Out[10]:
    a    1
    b    2
    c    3
    d    4
    dtype: int64
```

## Series using a numpy array

And finally, we can create a Series from a numpy array.

```py
In[11]:
    d = np.random.randn(5) # Create a numpy array of 5 random numbers
    series = pd.Series(d)
    series

Out[11]:
    0   -0.507243
    1   -0.115731
    2    0.526216
    3   -0.495840
    4    0.065357
    dtype: float64
```

Series is also similar to numpy array in functionalities. We can index data from a Series object as we do from a numpy array.

```py
In[12]
    series[:3] # Returns first 3 elements

Out[12]:
    0   -0.507243
    1   -0.115731
    2    0.526216
    dtype: float64

In[13]:
    series[series > 0] # Returns elements whose value is greater than 0

Out[13]:
    2    0.526216
    4    0.065357
    dtype: float64
```

Now, once we created a Series, we can apply many different methods on it to achieve different tasks. A few of them are as below. Read more about all the methods available on Series object at [pandas official documentation](https://pandas.pydata.org/pandas-docs/stable/reference/series.html).

```py
In[14]:
    series.size # Returns size of the series.
Out[14]:
    5

In[15]:
    series.shape # Returns shape of the series.
Out[15]:
    (5,)

In[16]:
    series.add(1) # Returns series with elements' value increased by 1
Out[16]:
    0    0.492757
    1    0.884269
    2    1.526216
    3    0.504160
    4    1.065357
    dtype: float64

In[17]:
    series.sub(1) # Returns series with elements' value decreased by 1
Out[17]:
    0   -1.507243
    1   -1.115731
    2   -0.473784
    3   -1.495840
    4   -0.934643
    dtype: float64

In[18]:
    series.mul(10) # Returns series with elements' value multiplied by 10
Out[18]:
    0   -5.072431
    1   -1.157307
    2    5.262160
    3   -4.958401
    4    0.653565
    dtype: float64

In[19]:
    series.div(10) # Returns series with elements' value divided by 10
Out[19]:
    0   -0.050724
    1   -0.011573
    2    0.052622
    3   -0.049584
    4    0.006536
    dtype: float64

In[20]:
    series.abs() # Returns series with absolute value of elements
Out[20]:
    0    0.507243
    1    0.115731
    2    0.526216
    3    0.495840
    4    0.065357
    dtype: float64

In[21]:
    series.max() # Returns maximum of all the elements.
Out[21]:
    0.5262159987471329

In[22]:
    series.min() # Returns minimum of all the elements.
Out[22]:
    -0.5072430551642492

In[23]:
    series.sum() # Returns sum of all the elements.
Out[23]:
    -0.5272413324692681

In[24]:
    series.mean() # Returns mean of the series.
Out[24]:
    -0.10544826649385361

In[25]:
    series.median() # Returns median of the series.
Out[25]:
    -0.1157306666767307

In[26]:
    series.std() # Returns standard deviation of the series.
Out[26]:
    0.43073327047221427

In[27]:
    series.drop(4) # Returns series with specified index removed.
Out[27]:
    0   -0.507243
    1   -0.115731
    2    0.526216
    3   -0.495840
    dtype: float64

In[28]:
    series
Out[28]:
    0   -0.507243
    1   -0.115731
    2    0.526216
    3   -0.495840
    4    0.065357
    dtype: float64
```

Look at the last code when we print the original series object, we get the elements same as we created, because all the operations that we applied on the series object gives us the view of the series and doesn't affect the original object.
Okay, so this is just a beginning, there are many more interesting things that we can do with pandas such as dropping null values or replacing them with some other values from data, visualize data using so many plots and charts, combine and merge two or more Series, create and manipulate Time Series, etc.

That's it for this post.In the next couple of posts, I'll be playing with some real world datasets and make some analysis on those data using Pandas.

Get full code notebook here : [Github](https://github.com/jamesshah/pandas-notebooks/blob/master/pandas-1.ipynb)

#### Thank you for reading. Happy coding.👨‍💻👩‍💻


Understanding Machine Learning From A Different Perspective

6 Elements Of Machine Learning - A Beginner's Guide to Machine Learning 


_Note : Content of this article including some images are taken from the Deep Learning Course by One Fourth Labs._

Before introducing the 6 elements of Machine Learning, let's understand **Why do we need Machine Learning?** Why not the old school expert systems where we used to come up with a solution and then code it in one of our favourite programming languages for the computer to understand it and then feeding it some input to get the desired result? A simple example of such expert systems is "Whether a patient has dengue or not?" Now, how do doctors decide that? They ask the patient "Do they have high fever and cold? Do thay have headache or rashes?" i.e. Whether the patient has symptoms of Dengues or not? And from the past experience doctor decides "If the patient has Dengue or not?"

Now, How do we solve this problem using a computer? We can simply write some if-else conditions to get the output as shown below.

```python
if HIGH_FEVER and COLD:
    return no_dengue
elif HIGH_FEVER and VOMITING and not RASH:
    return no_dengue
elif HIGH_FEVER and VOMITING and RASH:
    return dengue
```

And now we can simply run this program and feed it with the symptoms of a new patient and can decide "Whether he/she has dengue or not?"
Cool, right? So _Why do we need Machine Learning?_

The answer is the limitations of the expert systems that we talked about. Predicting dengue or not was somewhat easy task as we know about it's symptoms but let's say we want to predict whether a patient has Ebola or not? In this case, **the symptoms(rules to check) are unknown** and thus we can't feed the computer the rules that we don't know.

Another limitations is that sometimes we have lots of data. For example, "Should a bank give you a loan or not?" Now, giving loan depends on lots of factors including your salary, your credit score, size of your family and many more. Now in this case, we have **so much data to make sense from** that writing if-else blocks is not an easy and effective task.

And sometimes, **there are rules that we can't express to the computer** to understand. For example, sometimes your salary may be a little less than the required salary to get loan but you've been an honest and a regular customer to the bank and the bank manager knows that though your salary is not enough to meet the minimum requirements but they believe you and your honesty and gives you loan anyway.How do you feed such rule to the computer? How can a computer understand honesty?

These are the limitations that needs to be conquered and Machine Learning Does exactly that.
Using Machine Learning, we give some data as input and some matching output for that data and then find the relationship between the input data and output data.
The 6 elements of the the Machine Learning are:

- [1. Data](#1-data)
- [2. Task](#2-task)
- [3. Model](#3-model)
- [4. Loss Function](#4-loss-function)
- [5. Learning Algorithm](#5-learning-algorithm)
- [6. Evaluation](#6-evaluation)
      - [Thank you for reading! Hope you liked it. #HappyLearning](#thank-you-for-reading-hope-you-liked-it-happylearning)

![6 Elements Of Machine Learning](https://res.cloudinary.com/bytetales/image/upload/q_auto/v1591177461/6-elements.webp)

Now, let's dive deep into these elements one by one and see how we can see Machine Learning through the lens of these elements.

---

## 1. Data

Data simply means information. All types and formats of information. Today,there is enormous amount of data produced every second and this data can be used to answer so many questions. There is text data as well as audio-video data, there is structured data as well as unstructured data.

![Abundant Data](https://res.cloudinary.com/bytetales/image/upload/q_auto/v1591177235/abundant_data.webp)

> Data is the fuel of Machine Learning.

When starting a Machine Learning project, we need some answers to the questions like,
"Do we have the required data? If no, how do we get the data and where from? And if yes, is it in a structured form or unstructured and raw?"

These questions lead us to different paths in the project. If we don't have the data needed, then Data Curation comes in picture. There are many ways to get the data.

_If you are lucky enough_, there are many resources where you can get datasets that are publically available. Few examples of such resources are [Google Dataset Search](https://datasetsearch.research.google.com/), [Kaggle Datasets](https://www.kaggle.com/datasets), [Indian Government Data](https://data.gov.in/).

_If you are rich enough_, there are many websites on which you can assign some people to curate and collect data for your needs. For example, let's say you have images of animals and you want these images to be labelled with the name of animal, then you can assign such task to people and you can get the data easily and in a short span of time. Few examples of such crowdsourcing marketplaces are
[Amazon mturk](https://www.mturk.com/), [Dataturks](https://dataturks.com/).

And _If you are smart enough_, you can conduct surveys and get data from people or using the data scrapping techniques you can collect data from different websites.

One important thing to remember is that it doesn't matter in which format you get the data, at the end all the data needs to be encoded as numbers before feeding it to the computers. For example, images can be converted into matrix of pixel values.
![Data As Numbers](https://res.cloudinary.com/bytetales/image/upload/q_auto/v1591178117/data_as_numbers_hqci2f.webp)

---

## 2. Task

Now, you have data. The next question you should ask is "What task do I need to achieve? Or What task can I achieve using the data that I have?"

Generally, Machine Learning tasks are divided into two parts: **Supervised Learning And Unsupervised Learning.** Supervised Learning is when you have some input data as well as some output data corresponding to the input data. Then you can use Machine Learning to get the relationship between the input and output data and then use that relationship to predict output for the new input. And Unsupervised Learning is when you have only input data.

![Supervised vs Unsupervised Learning](https://res.cloudinary.com/bytetales/image/upload/q_auto/v1591176864/ml_tasks.webp)

Supervised learning is further divided into two sub sections as **Regression and Classification** where Regression means you want to predict some continous value for example House Price Prediction and Classification means you want a discrete answer such as yes or no. An example of Classification is, given an image you want to detect whether there is a dog or not in that image.

![Regression Task](https://res.cloudinary.com/bytetales/image/upload/q_auto/v1591177124/regression.webp)

Unsupervised learning is often used for **Generation and Clustering**. For example, the image given below shows that given the tweets of US President Donald Trump as an input, machine generates more such tweets.

![Generation Task](https://res.cloudinary.com/bytetales/image/upload/q_auto/v1591176983/generation.webp)

Although Unsupervised learning is useful in some cases, most of the real world problems are solved using Supervised learning as it gives more accurate results than Unsupervised learning.

> "Supervised learning has created 99% of economic value in AI." - Andrew NG, Co-founder of Coursera.

---

## 3. Model

A model is nothing but _a mathematical function_ which defines relationship between input data and output data.

A model can be a simple Linear Function such as `y = mx + c` or it can be a complex function such as `y = sin(cos(ax)) + tan(sin(bx^2 + cx))` .

The example that we talked about before, given an image(_data_), predict if it has a dog or not(_task_), we have to build a system that answers this question and such system is called a _model_.

There are many models available that are created over the years and each one has it's own pros and cons and each one is suitable for a unique task. For a human, it is almost impossible to come up with a function that defines relationship between the input and output, given that the input data has over thousand different features.

![Models](https://res.cloudinary.com/bytetales/image/upload/q_auto/v1591176698/models.webp)

---

## 4. Loss Function

Although, it's not possible to come up with a _model_ just by looking at the given data, but let's assume, you have come up with a model that you think correctly defines the relationship between input and output. But, two of your friends are solving the same problem and they have come up with a slightly different _model_, then how do you decide, **which model is better?** Loss function gives the answer to that question.

![Which Function Is Better?](https://res.cloudinary.com/bytetales/image/upload/q_auto/v1591176134/which_model.webp)

As shown in the image above, three different values predicted by the models that you and your friends have written are `f1(x), f2(x) and f3(x)`, respectively. And the true value is `y`. Now, to know which model is best suited for our data, we compute _the difference between the true value and predicted value_ for all the data using a loss function. One such loss function is Square Error Loss.

![Square Error Loss](https://res.cloudinary.com/bytetales/image/upload/q_auto/v1591176528/square_error.webp)

And counting loss function value for all three models, we get to know that the best model for our data is the one with the minimum value of loss function which is `1.38`.

![Loss Function](https://res.cloudinary.com/bytetales/image/upload/q_auto/v1591176310/loss_function.webp)

This way using a loss function, we can get the best model that defines the relationship between our input data and output data. More such examples of Loss Function are **Cross Entropy Loss** and **KL Divergence**.

---

## 5. Learning Algorithm

Now, we assumed a hypothetical situation where you and your friends came up with a model for our data and task but that won't happen in reality as there will be hundred and thousands of features in the data and you can't just look at it and propose a model that satisfies the relationship. Instead, you can propose a function such as `y = ax^3 + bx^2 + cx` and say that I think this should define the relationship between the **input x** and **output y**, but you don't know the paramteres a, b and c. So, to identify these parameters for a model, we need a learning algorithm.
![Learning Algorithm](https://res.cloudinary.com/bytetales/image/upload/q_auto/v1591175871/learning_algorithm.webp)

As shown in the picture above, we have a model with three parameters a,b and c and we need values of **a,b and c** at which the loss function's value is minimum. In calculus, there are many optimization strategies available that we can use to find the best values of parameters which minimizes the loss.
Some examples of such optimization solvers are Gradient Descent, Adagrad, RMSProp, etc.

_Learning algorithm is one task that computer does, other than that from data collection to deciding task and then proposing a suitable model is done by humans but then finding parameters of the proposed model is done by computer._

---

## 6. Evaluation

Okay,let's say using the above 5 elements, you've created an ML model to detect the animal in the image. And you want to sell it to some company or just show off to your friends. How do you know that your model is detecting the right animal? You guessed it right! you have to test your model. That's what evaluation is. Testing your model by feeding it some test data and then checking if it predicts the correct output in all the cases? If not, then how accurate your model is?

_Note: You evaluate your model using test data and not the training data, but why so? that's beyond the scope of this article!_

There are many performance matrices available for evaluation, one such matrix is **accuracy**. We find the accuracy of a model `diving "Number of correct predictions" by "Total numbers of predictions"` , as shown in the image below.

![accuracy](https://res.cloudinary.com/bytetales/image/upload/q_auto/v1591175490/accuracy.webp)

---

##### Thank you for reading! Hope you liked it. #HappyLearning


Learning Data Structures And Algorithms. Motivation. Resources. Planning.

Learning Data Structures And Algorithms

Data Structures and Algorithms are like backbone of a well designed and efficient code. For a software developer, Data Structures and Algorithms are the most important assets for programming, but Algorithms are not only limited to software engineering but in any field, Algorithms are a great way to define instructions to achieve some specific task. As a software development student, I've always tried to learn Data Structures and Algorithms, and I know basics and some intermediate level stuff, but I've always wanted to learn more about it and I tried so many times but I lacked consistency. But in this quarantine period, I had all the time and decided to start learning Data Structures And Algorithms.

![Dev Prod](https://res.cloudinary.com/bytetales/image/upload/q_auto/v1589549483/dev_productivity_x6g8k1.webp)


## What is Data Structures and Algorithms?

**Data Structure** is a way to store and organize the data in order to facilitate access and modification to the data. There many different data structures such as Stacks , Linked List, Queues, Trees, Graphs, Hash Tables, etc.
And there are some programming language specific data structures,too. For example, Python provides Dictionaries, Tuples, etc.

**Algorithms** have many different definitions all around the world. In general terms, am Algorithm is a collection of explicit, precise and unambiguous instructions, implemented in defined sequence to accomplish a certain task. And thus, Algorithms are not limited to software development but it has many other uses, too. There are so many problems that can be solved using specific algorithms such as simple Sorting, Searching and also some complex problems such as Identifying all the genes in a human DNA, Predictions and Recommendation Systems.

## Motivation To Learn

Well as we all know, Data Structures and Algorithms are very important to crack coding interviews but that isn't the only thing where learning Data Structures and Algorithms are useful. Data Structures and Algorithms Knowledge give us the ability to improve our solution to the problem and the ability to write much better and efficient code.

But most importantly, it helps to build problem solving mindset. And programming is almost all about problem solving. We can adapt any programming language syntax but the problem solving skill can't be adapted overnight. Thus, learning Data Structures and Algorithms can be a major learning curve for any computer science student.

## What I Have To Learn?

1. Algorithm Design
2. Algorithm Analysis
3. Algorithm Techniques
4. How to improve the time and space complexity
5. Advanced Data Structures (Trees, Graphs,etc.)

## My Resources & Plan To Learn Data Structures and Algorithms

There are so many books, courses, blogs and videos to learn Data Structures and Algorithms. And every DSA learner has this question that "Which course/book is the best to learn DSA?" And I am no exception. I had the same question as well, but over the past 3 years, I've learned many programming languages and frameworks using online courses and books. And what I've realized is that You don't have to find the best resources.<mark> Just take a book or a course and start learning. </mark>

While learning anything, I start with an online course and then refer to some book. So I started a [Data Structures and Algorithms Specialization](https://www.coursera.org/specializations/data-structures-algorithms) by University Of California, San Diego on coursera. It consists total of 6 courses from Basic Data Structures and Algorithms to Advance Algorithms and Complexity. And in the last course they have a project "Genome Assembly Programming Challenge" . They also have a book to refer during the course which can be of great use if you prefer text based content over videos. I'm almost halfway through the first course of the specialization and What I like the most about this specialization is that all the courses have a lots of programming challenges as assignments to solve and so, you don't just learn theory but also, implement it in code.

Along with this course, I'm using [Introduction To Algorithms](https://amzn.to/2zJbd3K) book (aka CLRS). It's a great book for someone who knows basics of Data Structures and Algorithms but for a total beginner, it can be overwhelming. In my opinion, if you are starting from scratch you should refer to GeeksForGeeks for basic Data Structures and try to implement them in a programming language of your choice.

Also, Mathematics plays very important role in learning Data Structures and Algorithms. I have a pretty good background in Mathematics but still, I have to go through some topics of Algebra and Calculus to refresh the dead cells of my brain.😅

### Other Resources :

#### Courses : 
1. [Algorithms by Standford University](https://www.coursera.org/specializations/algorithms)

2. [DSA by GeeksForGeeks](https://practice.geeksforgeeks.org/courses/dsa-online)

3. [MIT 6.006 On Youtube](https://www.youtube.com/watch?v=HtSuA80QTyo&list=PLUl4u3cNGP61Oq3tWYp6V_F-5jb5L2iHb)

#### Books :
1. [Algortithms](https://amzn.to/2T8858e) By Robert Sedgewick

2. [Data Structures and Algorithms Made Easy](https://amzn.to/3bD1BVc) by Narasimha Karumanchi

#### Youtube Channels

1. [Codechef Official](https://www.youtube.com/user/codechefofficial)

2. [code_report]
(https://www.youtube.com/channel/UC1kBxkk2bcG78YBX7LMl9pQ/)

3. [GeeksForGeeks Official channel](https://www.youtube.com/channel/UC0RhatS1pyxInC00YKjjBqQ)


## How to be consistent ?

Learning Data Structures and Algorithms can be overwhelming sometimes but consistency is very important. The skills will fade away if you stop learning. I'll tell you what worked for me to be consistent. If you are stuck at some problem for a long time and by long time I mean hours. Then stop trying to solve it at that time because your mind won't be able to think about the solution at that time and eventually you will get frustrated and disappointed. But don't skip the problem, if you are not able to solve it after many attempts and you feel like skipping it, go to google and search for the solution and analyze your mistake and your thinking process and then close that solution and implement it by yourself. But, don't make it a habit. Be patient. Data Structures and Algorithms is a vast ocean and you can't learn it all in a day. Keep learning, keep making mistakes, and keep solving them.😉

## What's Next ?

Obviously, one cannot master Data Structures and Algorithms with just one book and a specialization course. It's a huge field and this is just a beginning. After this course, I'll be practicing my Data Structures And Algorithms skills on platforms like [Leetcode](https://leetcode.com/) and [Hackerrank](https://hackerrank.com). Also, some of my friends have recommended me this book called [Crack Coding Interview](https://amzn.to/2ZipqiT) by Gayle Laakmann McDowell, so I'll solve some problems from that book, too.

Also, if you have any suggestions, then please message me on this email. [jamesshah@bytetales.co](mailto:jamesshah@bytetales.co)   Or message me on [LinkedIn](https://www.linkedin.com/in/james-shah/).

___
##### Thank You For Reading. Happy Coding👩‍💻 👨‍💻 

How Anne Frank fought with all the odds and survived in hiding for 2 years and what we can learn from her during this quarantine time.

5 Things I Learned From "The Diary Of A Young Girl" by Anne Frank 


In 1942 when Nazis occupied Holland, a thirteen-year-old girl _Anne Frank_ along with her family went into hiding and lived cloistered in the "Secret Annex" of an old office building with Van Daan family to their company. The diary of a young girl is a collection of Anne Frank's daily thoughts that she wrote in the form of letters to her diary - "Kitty". Anne Frank vividly expressed her daily life in the attic with seven other people. From quarrels to laughter, from books to radio, and how she hides the pain and tears and anger behind her always-smiling, all-knowing face and living under the ever-present threat of getting caught and of death, she wrote about each and every thing gallantly. It's a heartbreaking story of a young girl who bravely survived 2 years in the hiding before someone betrayed the her and her family and gave their whereabouts to the Gestapo.

I just completed this book yesterday and I feel so good that I did. It was on my reading list for a long time and now I finally read it in the quarantine period. And this is the most relatable book in the current time and the best book that I've read in the quarantine period. By that, I don't by any means want to compare the current situation of the world with that of the second world war's time but the quarantine period has got us locked down in our homes and we don't know when it's going to end.

![](https://res.cloudinary.com/bytetales/image/upload/c_scale,w_200,h_300,dpr_auto/v1588846379/the-diary-of-a-young-girl_anf9oq.jpg)

The way Anne Frank, _who didn't want to be treated as a girl-like-all other, but Anne-on-her-own-merit_ handles the changes that happen around her and also within her body is an inspiring story and there are many more things that we can learn from this book that could be of use in this quarantine period. The way she writes about her loneliness in her Dear Diary - Kitty, even though she has lovely and supporting parents who have spoiled her with their love and pampering and the way she doesn't hesitate to stand up for herself and not only others' but the way she addressed her flaws and faults is a great lesson, one can learn from this book.

---

Here are some of the my favorite quotes from the book and what we can learn from them in this quarantine period.

## "If we can save someone, then everything else is of secondary importance."

Anne Frank loved her father, Otto Frank more than anyone. He always stood by her in her decisions and supported her and spoiled her with his love and also he taught her many things, this quote is one of them. This teaching from Anne's father to her is probably the most important lesson that we should learn as the entire world is going through a lot of problems and pain and negativity. Helping each other is the only thing that we can do right now. Together, we can fight with this invisible enemy and win and get back to the normal routine of life.

## “I don't think of all the misery, but of the beauty that still remains.”

Anne Frank had to go through things that we can't even imagine to be in and yet she didn't get sad because of it all but on the contradiction, she thought about nature, about Peter, about what she will do _'When the war gets over'_ . With all the surrounding negativity, it is very natural to get depressed but we shouldn't be thinking about the misery but we should try to think of good things. You may be locked down for months and it's hard for you, me, and everyone to get stuck in our homes, but we should look at the brighter side, too. When was the last time, we spend so much time with our family or with ourselves. This may be a time we do something that we always wanted to but didn't have time for. We should think of this as an opportunity to change our life for the betterment.

## “Where there's hope, there's life. It fills us with fresh courage and makes us strong again.”

We shouldn't lose hope. Hope is all that we've got now and we can't afford to lose that. Let's hope this night of the misery will go soon and the bright days will return. Let's hope we lose no more lives of our fellows who are suffering from the corona-virus or any other diseases. Hope can set us free, but fear can hold us, prisoner. The way Anne and her family don't lose hope even when they face hunger, boredom, the ever-present threat of discovery and death, it's very inspiring to see them not losing it at all. Yes, they've had quarrels, they tired of looking at the same face all day for over two years but they still didn't lose hope. And that's what we should also do, not lose hope!
😇

## "A person who's happy will make others happy. A person who has courage and faith will never perish in misery."

Be Happy. It's very important to be happy.The people hiding in "Secret Annex" never missed a chance to do something that will bring joy and make them happy. They celebrated birthdays, they made jokes, they shared food and books and their stories from before the hiding. Some way or another, they managed to bring smile on each other's faces. Being sad will only bring more negative thought and more miser. This is the time when we should find the happiness in all the little things possible and spread happiness to others,too. Let it be a small message to your loved ones asking about their health or it may be a funny video or a happy book that you read and recommend others to read,too. You are brave and you should believe in yourself. Let's spread smiles and not corona-virus!
😉😉

## "I trust to luck and do nothing but work, hoping that all will end well."

This is my favorite quote from the book.Anne, Margot and Peter don't let their situation affect their learning and they read books and learn new languages. Not only them but their parents,too. And that's what we should do, we shouldn't make the lock down an excuse to not work or learn but we should take it as an opportunity to do things that we couldn't do because we didn't have enough time from our routine.
We should do what's in our hand and what's in our control, which is nothing but our work, and that's what we should do. We don't know when and how this pandemic is going to end and we can't do anything in that matter but what we can do is our work.
💪

---

Anne Frank taught all of us many many things through her diary. Her braveness, wittiness and maturity to handle the tough times at such a small age is commendable. I recommend this book to everyone, especially now than ever.
You can buy this book on kindle here:

[Buy Now On Kindle](https://amzn.to/2W9b81W)

##### Thank You For Reading! HappyQuarantine🤗


Let's write a Python script to generate Tambola Housie tickets.

Housie(Tambola) Ticket Generator using Python

Housie aka Tambola aka Bingo (British Version) is probably the best game to play in Lockdown to get our minds off from Corona and all the negative news from all around the world and to have fun with friends and family. I've been playing it a lot,lately.
For those who don't know what Housie is, let me explain.

# What is Housie? 
Housie is basically a game of probability in which players mark off numbers on cards as the numbers are drawn randomly by a caller. So at the beginning of the game, all the players gets a ticket [3x9 matrix] in which there are total 27 boxes and 15 of them are filled with a distinct random number from 1 to 90. Here is a sample housie ticket.

![Housie Ticket Example](https://res.cloudinary.com/bytetales/image/upload/w_200,c_scale/v1588228912/Bingo_Card_wjimgg.svg)

And then a caller calls random number from 1 to 90 and the winner being the first person to mark off all their numbers. So, It's pretty simple yet interesting and gripping game that one can play with friends and family. 

# Housie Ticket Rules

Now there are some rules to generate housie tickets and you can't just randomly put 15 numbers in any boxes of your choice. So, let's see the rules and then create a Python script that will generate such housie ticket for us.

**RULE #1** - Each row cannot have more than 5 numbers.

**RULE #2** - Each column is assigned a range of numbers only:   
(ex. 1-10 can appear only in column 1)

**RULE #3** - In a specific column, numbers must be arranged in ascending order from top to bottom.

Now, ofcourse it's a cumbersome task(and a little biased,too) for a human to generate such tickets, and for the record we are talking about atleast 20-25 tickets per game. So yeah it's definitely not a fun task. So, I thought of creating a python script that would take the number of tickets that we want to generate as input and will return the tickets as an output.

I know, it's not at all a very fancy or difficut script to write but anyway I am free in this lockdown period so I thought why not?

Let's see how we are gonna build this script.
***

### Step 1 : Create a 3x9 matrix (array) of zeros 
```py
import numpy as np

# Create a 2D array[3x9] of 0s
ticket_array = np.zeros((3, 9), dtype=int)
```

Here I've used `numpy` library of Python to create the `ticket_array` because while creating a single matrix, it may not be effective but when we want to create very large number of lists, numpy creates it very fast and therefore it is efficient to use numpy to create the array. 

### Step 2: Randomly Generate 15 indices to fill values in a ticket.
Now, in numpy we can use tuple for indexing of an array. i.e. To get the first element of first row and first column from `ticket_array` , we can use `ticket_array[(0,0)]`. 

And so, first we'll generate an array `total_indices` which contains all  the indices of 3x9 array as tuples (i.e. (0,0), (0,1)), (0,2), ..., (2,8)) 

and then, we'll randomly generate 5 indices from each row.
```py
total_indices = [(i, j) for i in range(3) for j in range(9)]

random_indices = []


first_row = random.sample(total_indices[:9], 5)
second_row = random.sample(total_indices[9:18], 5)
third_row = random.sample(total_indices[-9:], 5)

for i in first_row:
    random_indices.append(i)

for i in second_row:
    random_indices.append(i)

for i in third_row:
    random_indices.append(i)
```  

### Step 3: Now Let's fill this indices with random values

So let's create a list ranging from 1 to 90. Then to satisfy the Rule #2, we'll check the column number from index and will fill the according value.
```py
for num in random_indices:
    if num[1] == 0:
        number = random.choice(total_numbers[:10])
        ticket_array[num] = number
        total_numbers[total_numbers.index(number)] = 0
    elif num[1] == 1:
        number = random.choice(total_numbers[10:20])
        ticket_array[num] = number
        total_numbers[total_numbers.index(number)] = 0
    elif num[1] == 2:
        number = random.choice(total_numbers[20:30])
        ticket_array[num] = number
        total_numbers[total_numbers.index(number)] = 0
    elif num[1] == 3:
        number = random.choice(total_numbers[30:40])
        ticket_array[num] = number
        total_numbers[total_numbers.index(number)] = 0
    elif num[1] == 4:
        number = random.choice(total_numbers[40:50])
        ticket_array[num] = number
        total_numbers[total_numbers.index(number)] = 0
    elif num[1] == 5:
        number = random.choice(total_numbers[50:60])
        ticket_array[num] = number
        total_numbers[total_numbers.index(number)] = 0
    elif num[1] == 6:
        number = random.choice(total_numbers[60:70])
        ticket_array[num] = number
        total_numbers[total_numbers.index(number)] = 0
    elif num[1] == 7:
        number = random.choice(total_numbers[70:80])
        ticket_array[num] = number
        total_numbers[total_numbers.index(number)] = 0
    elif num[1] == 8:
        number = random.choice(total_numbers[80:89])
        ticket_array[num] = number
        total_numbers[total_numbers.index(number)] = 0
```

### Step 4: Now let's sort the ticket_array column wise

```py
for col in range(9):

    # if all the rows are filled with random number

    if(ticket_array[0][col] != 0 and ticket_array[1][col] != 0 and ticket_array[2][col] != 0):
        for row in range(2):
            if ticket_array[row][col] > ticket_array[row+1][col]:
                temp = ticket_array[row][col]
                ticket_array[row][col] = ticket_array[row+1][col]
                ticket_array[row+1][col] = temp


    # if 1st and 2nd row are filled by random number

    elif(ticket_array[0][col] != 0 and ticket_array[1][col] != 0 and ticket_array[2][col] == 0):
        if ticket_array[0][col] > ticket_array[1][col]:
            temp = ticket_array[0][col]
            ticket_array[0][col] = ticket_array[1][col]
            ticket_array[1][col] = temp


    # if 1st and 3rd row are filled by random number

    elif(ticket_array[0][col] != 0 and ticket_array[2][col] != 0 and ticket_array[1][col] == 0):
        if ticket_array[0][col] > ticket_array[2][col]:
            temp = ticket_array[0][col]
            ticket_array[0][col] = ticket_array[2][col]
            ticket_array[2][col] = temp

    # if 2nd and 3rd rows are filled with random numbers

    elif(ticket_array[0][col] == 0 and ticket_array[1][col] != 0 and ticket_array[2][col] != 0):
        if ticket_array[1][col] > ticket_array[2][col]:
            temp = ticket_array[1][col]
            ticket_array[1][col] = ticket_array[2][col]
            ticket_array[2][col] = temp
```

### Step 5: Wrap it up in a function and print the tickets.

To print the tickets I used tabulate package of python using which we can present array and matrix in a neat way. Learn more about Tabulate [Here](https://pypi.org/project/tabulate/).

```
if __name__ == "__main__":

    # Take number of tickets from user as system argument
    numberOfTickets = sys.argv[1]
    tickets = []

    for i in range(int(numberOfTickets)):
        ticket = getTickets()
        tickets.append(ticket)

    for ticket in tickets:
        print(tabulate(ticket, tablefmt="fancy_grid", numalign="center"))
```    
And Done!! Now, Let's check by running the script from terminal.

```sh
$ python ticket-generator.py 2 
```

**Output:**

![Output](https://res.cloudinary.com/bytetales/image/upload/w_500,c_scale/f_webp/v1588236182/ticket_example_zm5gpl.webp)

___
You can checkout the full code on Github : 
[Housie Ticket Generator](https://github.com/jamesshah/housie-tickets-generator)

Thank You For Reading!
##### HappyCoding👨‍💻👩‍💻

Scheduling Daily SMS using Cron Jobs And Deploying On Heroku.

Building A Node.js - Express App (covid19-updates) | Part-4

_If you haven't read the last post, I recommend you to check it out [here](https://www.bytetales.co/building-a-node-js-express-app-covid19-updates-part-3/), else many things in this post will not make sense._

In the last post we finally completed the database creation and user registration and also scheduled twilio sms api to send the case updates sms to the users daily and now in this post we will receive an sms from the user, in case they want to stop the service and we'll deactivate their account to stop sending messages to them anymore and will send them a reply back. And after that, we'll deploy our webapp on [Heroku](https://www.heroku.com/). Although, there are many cloud services available to serve Node.JS application online but I chose heroku as it's makes deployment very easy using github integration. Let's start with receiving an SMS from the user.
***
## Receiving and Replying to SMS using Twilio

Twilio can send our web application an HTTP request when certain events happen, such as an incoming text message to our Twilio phone number.So, we'll create a route that twilio will call whenever a user sends a message on our Twilio phone number to stop or restart the service.
Now, We need to configure our Twilio phone number so that it can send an HTTP request to the route in our application, whenever our it receieves an incoming message. But, we'll need our websites URL for that,which we don't have for now. So, let's skip this and create the route in our application and after we deploy our application on heroku, we'll add the URL in the Twilio console.

Create a new file in `routes` directory called `sms.js` and add the following code to it.
_Twilio provides code snippet for almost every programming language to achieve this task._

```js
const express = require("express");
const router = express.Router();
const MessagingResponse = require("twilio").twiml.MessagingResponse;
const User = require("../models/User");

router.post("/", async (req, res) => {
  const twiml = new MessagingResponse();
  phn_number = req.body.From.slice(3);

  if (req.body.Body === "STOP") {
    await User.update(
      { number: phn_number },
      {
        $set: {
          active: false,
        },
      }
    );
    twiml.message(
      "You will not receive updates any more. To restart the service reply with 'START'\n Thank You."
    );
  } else if (req.body.Body === "START") {
    await User.update(
      { number: phn_number },
      {
        $set: {
          active: true,
        },
      }
    );
    twiml.message("Thank you for your interest. You will now receive updates.");
  } else {
    twiml.message("Sorry, you've send an invalid response.");
  }
  res.writeHead(200, { "Content-Type": "text/xml" });
  res.end(twiml.toString());
});

module.exports = router;
```
So, now when a user sends an sms to our Twilio phone number with 'STOP' or 'START' in the message body, Twilio will call the `/sms` route in our application and our application will handle the sms according to the message body and will reply to the user.
We'll need to add the following code in `app.js` to add this route in our application.
```js
app.use("/sms", require("./routes/sms"));
``` 

And we are done with the coding part. Now, let's deploy the application on Heroku and set it live for the users to use.I will be deploying the app on Heroku using Github repository, however we can also make use of `heroku-cli` to deploy. You can learn more about Heroku CLI at their website, [here](https://devcenter.heroku.com/articles/heroku-cli).
***
## Creating a Github Repository And Deploying using Heroku

Before creating a Github repository, we'll add a `.gitignore` file in our project directory to specify what parts of our application we want to ignore(i.e, node modules, .env file) while uploading it to the Github repository. We can manually write the `.gitignore` file but for the best practice we can get the specific file for specific programming language or framework that we use in our application. Github itself provides a repository that contains the collection of such files that we can use in our application and I used one for this application as well. You can find it [here](https://github.com/github/gitignore/blob/master/Node.gitignore).

Now, let's create a new Github Repository and use our local project directory to upload.If you don't know how to create a github repository, you can learn how to do it from their [documentation](https://help.github.com/en/github/getting-started-with-github/create-a-repo).
![New Github Repo](https://dev-to-uploads.s3.amazonaws.com/i/b8ss1vhouitztg4kk4de.png)

Now, let's go to Heroku and sign in to your account, create a new account,if you don't have one by simplying filling out a form for free. 
![New Heroku Account](https://dev-to-uploads.s3.amazonaws.com/i/c4u9cpqb2f2rs7jxu61j.png)

Now, on the dashboard, let's creat a new app and then connect your Github account on the next page and select the Project Repository that we've just created and connect.

![Github Heroku Connect](https://dev-to-uploads.s3.amazonaws.com/i/afpsu3a2ae3gyjtwvjlp.png)

Now your app has been created but to tell Heroku that this is a Node.JS app, we'll need to add a buildpack. Go to settings and select "Add Buildpack" tab and select **nodejs** option from the list.

![Select Node.Js Buildpack](https://dev-to-uploads.s3.amazonaws.com/i/rddzc60tzj9yvjruwd2y.png)

Remember, we used `.env` file to hide our secrets but we've not uploaded it to our Github repository for the very obvious reason. So let's add those secrets in **Heroku Config Variables**. In settings tab select Reveal Config Var and then add all the keys and values one by one.

![Heroku Confing Var](https://dev-to-uploads.s3.amazonaws.com/i/druwipcwrmqmartfwhh1.png)

And that's it, Now we can visit the url by clicking **Open App** from dashboard and we have our app live on server!!

![WebApp Live On Server](https://dev-to-uploads.s3.amazonaws.com/i/p67a56ruhp38o8v6mvc6.png)

Now remember, we skipped a step above, configuring the Twilio console to add the URL of our app. Let's finish it by going into console page of our Twilio Account. Then click on your Twilio phone number. Find the Messaging section and the "A MESSAGE COMES IN" option and add the url to the application and add `/sms` as it is the route that handles incoming messages in our application.

![Twilio Console](https://dev-to-uploads.s3.amazonaws.com/i/jog4f9eopgkp5ov7u8qk.png)
And Finally, We've completed all our project from the scratch to find an idea, learning tech stack, creating and then deploying the application.
***
Full code can be found here: 
[Covid-19 Updates India](https://github.com/jamesshah/Covid19-Updates)

Thank You For Reading. 
##### HappyCoding👩‍💻 👨‍💻


Let's get data from external API and use Twilio API to send COVID19 case updates to the users.

Building A Node.js - Express App (covid19-updates) | Part-3

_If you haven't read the last post, I recommend you to check it out [here](https://www.bytetales.co/building-a-node-js-express-app-covid19-updates-part-2/), else many things in this post will not make sense._

In the previous post, we created database and user registration for our application and now in this post we'll add Twilio API to send SMS to users whenever they register successfully and we'll also add external API call to get the coronavirus data to send to our users. Lots of work to do!! So, let's get started.

***
## Using Twilio API

[Twilio](https://www.twilio.com) provides many [services](https://www.twilio.com/docs) ranging from SMS and Chat to AI Bots, Programmable SMS API being one of them. Using the Programmable Twilio SMS API we can add SMS service in our web application and that's exactly what we'll doing.

To use the Twilio SMS API,we need to create an account on Twilio website and buy a twilio number with SMS support. Twilio provides free number with some limitations. But luckily, As a part of #DevTwilioHackathon they are providing Free 20\$ credits using the promo code `DEVHACK20`. So I went ahead and bought a number with this promo code. And now if we go to the console on the twilio website, we can get 'ACCOUNT SID' and 'AUTH TOKEN' which will be used to connect to the Twilio API in future. Copy that and paste it into the `.env` file in our project directory as we don't want anyone to be able to see it.


![Twilio Console](https://dev-to-uploads.s3.amazonaws.com/i/j966mkml979f0boqbbt4.jpg)

To use the twilio API in our application, we'll need Twilio Helper Library and we can download it using the following code in the terminal.

```bash
$ npm install twilio
```

Now, Twilio provides code snippets to connect to it's various APIs in many different programming language, And we'll use it to connect to the twilio api in our app.Let's go to the `users.js` file in our project directory and add some code to connect to the API.

```javascript
const accountSid = process.env.ACCOUNT_SID;
const authToken = process.env.AUTH_TOKEN;
const client = require('twilio')(accountSid, authToken);

client.messages
  .create({
     body: 'Thank you for your registration. You will receive Corona-Virus updates everyday.To stop the service reply with "STOP"',
     from: process.env.PHN_NUM,
     to: // user's mobile number (with country code).
   })
  .then(message => console.log(message.sid));
```

We'll add this snippet in the `/users/add_user` route, so that whenever a new user gets added to the database, we'll send the SMS to the user. Here is the sample SMS that our application sends to the user.

![Registered Successfully](https://dev-to-uploads.s3.amazonaws.com/i/ojlcdyiwxesjyhm5q998.jpg)
***
## Getting Data From The External API

Now, we want to send corona virus case updates to our users but where do we get the data from? And there comes the External API. There are so many Corona Virus datasets available publically online. I've used the [api.covid19india.org](https://api.covid19india.org) to get the data as it is crowdsourced and it is based on the data declared by the State Governments so it's a reliable source of data and also, it gets updated every 5 minutes, so we'll get latest updates.And also, this api requires no authentication at all.

In node.js, we cannot use `fetch` as we do in the client-side javascript, but there is an npm package to our rescue called `node-fetch`. So let's install it.

```bash
$ npm install node-fetch
```

Now, let's add a new file in our project directory called script.js and let's add some code in it to get the data from the api and use it.

```javascript
const fetch = require("node-fetch");

const get_data = async () => {
  cases = {};
  await fetch("https://api.covid19india.org/data.json")
    .then((res) => {
      return res.json();
    })
    .then((data) => {
      data_array = data.statewise;
      total_obj = data_array.filter((data) => data.state === "Total")[0];
      gj_obj = data_array.filter((data) => data.state === "Gujarat")[0];
      mh_obj = data_array.filter((data) => data.state === "Maharashtra")[0];

      cases.total_cases = total_obj.confirmed;
      cases.total_new = total_obj.deltaconfirmed;
      cases.mh_total = mh_obj.confirmed;
      cases.mh_new = mh_obj.deltaconfirmed;
      cases.gj_total = gj_obj.confirmed;
      cases.gj_new = gj_obj.deltaconfirmed;
    })
    .then();
  return cases;
};
```

Okay, let me explain it a little bit. Here, we are fetching the state wise data from the api and it returns array of objects in which there are many properties like, total cases, new cases, total deaths, new deaths, total recovered,etc. We want to find the total cases and new cases in states. Here i've shown only 2 states (Gujarat & Maharashtra) but we can do the same for any state and can get the same data for all the states of India. So i've created an object called 'cases' and in this object I've added 6 keys namely, 'total_cases', 'total_new', 'mh_total', 'mh_new', 'gj_total', 'gj_new' and their values gives the number of 'Total Cases in India', 'New Cases in India', 'Total Cases in Maharashtra','New Cases in Maharashtra', 'Total Cases in Gujarat', 'New Cases in Gujarat' respectively.
***
## Sending Message To The Users According To Their State

We want to send a distinct message to the user according to his/her state.i.e, if the user registered with the state 'Gujarat' then we want to send updates of cases in Gujarat and in India and for users of Maharashtra, we want to send updates of cases in Maharashtra and in India. So to do that we'll create two different messages to send to the user and for all the users in our database we'll check the state and send the according message to them.Let's create a new function get_users to get the users mobile numbers and divide them into different arrays according to their state.

```javascript
const User = require("../models/User"); // To retrieve users from the Database.

const get_users = async () => {
  let numbers = {};

  // Getting all the active users from the Database
  const active_users = await User.find({
    active: true,
  });

  // Getting an array of numbers of Active Users in Gujarat
  guj_users_num = active_users
    .filter((user) => user.state === "Gujarat")
    .map((user) => user.number);

  // Getting an array of numbers of Active Users in Gujarat
  mh_users_num = active_users
    .filter((user) => user.state === "Maharashtra")
    .map((user) => user.number);

  numbers.MH = mh_users_num;
  numbers.GJ = guj_users_num;

  return numbers;
};
```

So, as we can see we first get all the active users from our database and insert their mobile numbers in different arrays and add that arrays to the 'numbers' object with the corresponding key.

Now, let's create a new function send_msg() to create and send message to our users using the Twilio API.

```javascript
const send_msg = async () => {
  // Getting Users' Mobile Numbers And Data From API.
  const users_num = await get_users();
  const cases_data = await get_data();

  // Message For Gujarat Users
  const gj_msg = `Total Cases In India: ${cases_data.total_cases} \n Total Cases In Gujarat: ${cases_data.gj_total} \n New Cases In India: ${cases_data.total_new} \n New Cases In Gujarat: ${cases_data.gj_new} \n #StayHome #StaySafe`;

  // Message For Maharashtra Users
  const mh_msg = `Total Cases In India: ${cases_data.total_cases} \n Total Cases In Maharashtra: ${cases_data.mh_total} \n New Cases In India: ${cases_data.total_new} \n New Cases In Maharashtra: ${cases_data.mh_new} \n #StayHome #StaySafe`;

  // Sending Messages To Users In Gujarat
  users_num.GJ.forEach((user) => {
    client.messages
      .create({
        body: gj_msg,
        from: process.env.PHN_NUM,
        to: "+91" + user,
      })
      .then((msg) => console.log(msg.sid))
      .catch((err) => console.log(err));
  });

  // Sending Messages To Users In Maharashtra
  users_num.MH.forEach((user) => {
    client.messages
      .create({
        body: mh_msg,
        from: process.env.PHN_NUM,
        to: "+91" + user,
      })
      .then((msg) => console.log(msg.sid))
      .catch((err) => console.log(err));
  });
};
```
***
## Scheduling Automatic SMS

We have our message ready with us to send to users but how do we schedule it to send automatically at a specific time? Cron Jobs! The `node-cron` package helps us schedule the job to perform at a specific time.So, let's install this npm package and use it to schedule our send_msg function to send SMS automatically everyday.

```bash
$ npm install node-cron
```

```javascript
const cron = require("node-cron");

const tasks = cron.schedule("0 9,20 * * *", () => {
  send_msg();

  modules.exports = tasks;
});
```

The schedule method of cron module takes two arguments, first the pattern of time that we want to schedule and second is the function that we want to schedule. I've set time to 9a.m. and 8p.m. Everyday.
[Read More About Cron Patterns Here](https://www.npmjs.com/package/node-cron).

And now we'll have to call the method `tasks.start()` in our `app.js` file when the server starts like this.

```javascript
const script = require("./script");

app.listen(process.env.PORT || 3000, () => {
  script.tasks.start();
});
```

And that's it.Now when we deploy our application on server, it will send sms to the users automatically everyday at 9 a.m. and 8 p.m. using the data retrieved by the external api. Here is what this message looks like.

![SMS Updates](https://dev-to-uploads.s3.amazonaws.com/i/sn4sisxrbsl7e7z9xggk.jpg)


And Bingo! We've created a fully functional application which registers users and sends them cases updates daily automatically. Now in the next post, we'll add functionality to receive sms to users in case they want to stop the service and then we'll deploy our application on heroku server.
***
Full Code Can Be Found Here:

[Covid-19 Updates India]
(https://github.com/jamesshah/Covid19-Updates)

Thank You For Reading. 

##### HappyCoding👩‍💻 👨‍💻


Connecting MongoDB Database and Creating User Registration in Express (Node.JS)

Building A Node.js - Express App (covid19-updates) | Part-2

_If you haven't read the last post, I recommend you to check it out [here](https://www.bytetales.co/building-a-node-js-express-app-covid19-updates-part-1/), else many things in this post will not make sense._

In the last post we've created a simple index('/') route with HTML file as response using EJS template engine.
Now, it's time to look into middlewares and how can we use them to serve our routes with the same prefix easily and create user registration.
___
## Middlewares

Middleware functions are functions that have access to the request object (req), the response object (res), and the next function in the application’s request-response cycle.The next function is a function in the Express router which, when invoked, executes the middleware succeeding the current middleware.

For example, let's say you have a route ('user/login') in your express app to login the user. Now you want to check the credentials given by users with the credentials stored in the database and authorize that user to the account.And when do you check this credentials? After the user sends request and before your application sends the response. And that's when the middlewares comes in handy. You can write your logic to authenticate the user in the middleware function and response to the user according the return value of the middleware function.Other than authentication, middlewares have many uses as well. [Read More About Middleware Here](https://expressjs.com/en/guide/writing-middleware.html).
 
To use the middleware function, we need to add `app.use(middlewareFunctionName);` in our code. And that will add the defined middleware function to all the routes that we define in our application. It is called the 'application level middleware'. But we can also add middlewares to particular routes of our choice by providing that route as the first argument in the app.use function. i.e : `app.use('/', middlewareFunctionName);`

Now, let's add middlewares in our code to serve the routes from routes directory.

```javascript
app.use('/', require('./routes/index'));
```

Now let's create the user registration page in our app. But that will require MongoDB database. So, I created an account on [MongoDB Atlas](https://www.mongodb.com/cloud/atlas), they provide 500MB of free database hosting. And I got the database URL from that which we'll be using to connect to our database. the url seems something like this.
`"mongodb+srv://<username>:<password>@cluster0-kmdpn.mongodb.net/test?retryWrites=true&w=majority`
___
## Creating Database Model and Connecting to The Database

First of all, let's use mongoose and the database URL and see if we can connect to our databse.We will need mongoose package and we'll use it's `connect` method to connect to our database. Let's add the below code to our `app.js` file.

```javascript
const mongoose = require("mongoose");

const DATABASE_URL = "mongodb+srv://<username>:<password>@cluster0-kmdpn.mongodb.net/test?retryWrites=true&w=majority";

mongoose.connect(DATABASE_URL, (err) => {
	if (err) console.log(err);
	else console.log("Connected to DB.");
})
```

Now we can start the server and see if we've connected to the database or not but we are currently working on a localhost server and this code is in our machine only but when we deploy our application online on cloud, we don't want everyone to see the DATABASE_URL as it contains our username and password of the database. So, to protect it we can use `.env` file to store our secret credentials.And while uploading it online we won't be adding this `.env` file. To use it let's install the ['dotenv' npm package](https://www.npmjs.com/package/dotenv).

```bash
$ npm install dotenv
```

Then,create a `.env` file in the project directory and add DATABASE_URL to this file. And to use the value from this file in our application we can make following changes.
```javascript
require("dotenv").config();

// To connect to MongoDB Database
mongoose.connect(
  process.env.DB_CONN,  
  (err) => {
    if (err) console.log(err);
    else console.log("Connected to DB!");
  }
);
```

Okay!! So now we have our application connected to our database.
Now,let's create a database model which is basically a schema of our user's information. So first let's create a new directory 'Models' in our app directory and create a new file `User.js`. In this file, we'll write the below code to create and export our model. 
We'll need 4 field in our User model. Name, Contact Number, State, and Active. In which, Name,Number and State fields will be String type and Active field will be Boolean type.

```javascript
const mongoose = require("mongoose");
const UserSchema = mongoose.Schema({
  name: {
    type: String,
    required: true,
  },
  number: {
    type: String,
    required: true,
  },
  state: {
    type: String,
    required: true,
  },
  active: {
    type: Boolean,
    required: true,
    default: true,
  },
});

module.exports = mongoose.model("User", UserSchema);
```

Now, we have our User model ready with us. Now, we we'll create routes and create HTML page to register a user. Let's start by creating ('/users/register') route. Create a new file `users.js` in routes directory and add the below code.

```javascript
const express = require("express");
const router = express.Router();

router.get('/register', (req,res) => {
	res.render('register');
});

module.exports = router;
```
___
## Creating User Registration Form and Registering Users

Here we've used an in-built express middleware Router. Using router instance, we can easily serve routes with same prefix which is 'user'. We'll be creating more routes with this 'user' prefix and therefore we can take advantage of the Router middleware.Now, let's create a form for users to register in the register view that we are rendering to the register route.

```html
<div class="jumbotron">
  <form action="/users/add_user" method="post">
    <fieldset>
      <legend
        style="text-align: center; text-decoration: underline;"
        class="mt-3"
      >
        Register Now
      </legend>
      <div class="form-group">
        <label for="name">Name</label>
        <input
          type="text"
          name="name"
          class="form-control"
          id="name"
          required="true"
          placeholder="Enter Your Name"
        />
      </div>
      <div class="form-group">
        <label for="phone">Phone Number</label>
        <input
          name="number"
          type="text"
          class="form-control"
          id="number"
          required="true"
          aria-describedby="phoneHelp"
          placeholder="Enter Your Number"
        />
        <small id="phoneHelp" class="form-text text-muted"
          >We'll never share your number with anyone else. Promise!</small
        >
      </div>
      <div class="form-group">
        <label for="state">State</label>
        <select
          class="form-control"
          id="state"
          name="state"
          style="cursor: pointer;"
          required="true"
        >
          <option
            selected="selected"
            id="state_default"
            value=""
            style="display: none;"
            >Select State</option
          >
          <option value="Gujarat">Gujarat</option>
          <option value="Maharashtra">Maharashtra</option>
        </select>
      </div>
      <button type="submit" class="btn btn-primary">Count Me In!</button>
    </fieldset>
  </form>
</div>
```
It should look something like this,
![Registration Form](https://dev-to-uploads.s3.amazonaws.com/i/bbknovnwiu3wj2lneku3.png)

As we've added the path `/users/add_user` in the action attribute of the form tag, let's create that route in our `users.js` file to register user to our database.

In this, first we'll check if the phone number entered by the user already exist in our database, if it exists we'll redirect the user to the `already_registerd` page and if it doesn't exist already, we'll add it our database and redirect the user to the `register/successful` page. This is going to be a little bit complex, let's start coding.

```javascript
const User = require("../models/User"); // To access the User Model

router.post("/add_user", async (req, res) => {
  try {  	
    const { name, number, state } = req.body; // Getting Form Fields From The Request Body.    
    const new_user = await User.aggregate([{ $match: { number: number } }]); // Checking if the number already exists.
    if (new_user.length != 0) {
      res.render("already_registered",
        {
          message: "It Seems Like, This number is already registered with us.",          
        }
      );
    } else {
    	// Create a new user object to add to the DB.
      const user = new User({
        name: name,
        number: number,
        state: state,
      });
      const addedUser = await user.save(); // Saving the user to the database.
      res.redirect("/users/register/successful");
  	}
  } catch (error) {
    console.log(error);
    res.json({ message: error });
  	}
};
```

And Done!! Really?? No, This code will generate an error and that's because we can't directly use the request body in our app. We'll have to convert it to JSON to be able to use it in our application. And to do that we'll use `body-parser` package. Let's add that in `app.js` and we are done with the registration!!

```bash
$ npm install body-parser;
```

```javascript
const bodyParser = require("body-parser");

app.use(bodyParser.urlencoded({ extended: false }));
app.use(bodyParser.json());
```

And now we are finally done!! Now in the next post we'll add the most exciting part of this application - the Twilio API and we'll send the users an SMS when they register to let them know they have registered successfully and also we'll create a route to receive the message from the users, if they want to stop our service. Fun things are coming up!!

Full code can be found at this repository : 

[Covid19 Updates India]
(https://github.com/jamesshah/covid19-updates)

Thank you for reading!! 

##### HappyCoding👩‍💻 👨‍💻

Let's Learn Node.JS & Express To Build A Web App For a Hackathon

Building A Node.js - Express App (covid19-updates) | Part-1

## Brainstorming

In first week of April, Dev.to announced a [hackathon](https://dev.to/devteam/announcing-the-twilio-hackathon-on-dev-2lh8) in collaboration with Twilio to make a web app using any of the Twilio APIs, So I decided to participate and build something in this quarantine time which will help me learn some new tech and build a project to showcase.

So I started brainstorming for an idea to create an app for and just then I heard my mother saying that we are privileged to have an Internet service which helps us getting latest news and updates in this Covid-19 lockdown period but for those who doesn't have this privilege, It's hard to get information about new cases in their state as well as in the country.And bingo!! I got the idea to create my web app.

I decided to create a web app in which user can register with their phone number once and they can get daily updates of new cases in their area as well as in the country with the help of [Twilio Programmable SMS API](https://www.twilio.com/docs/sms).
___
## Deciding Tech-Stack

Now it was time to choose the Tech-Stack that i will be using to create this web app.I've created web apps using Django and Flask(Python Web Frameworks) in the past,and I am pretty much used to creating backend in Python but I wanted to learn new Stack and so I decided to take on Node.JS & [Express Framework](https://expressjs.com/) for this web app and for database, I've used SQL Databases (MySql, PostGreSql) in past and I wanted to learn NoSql Databases, so I decided to learn [MongoDB](https://www.mongodb.com) by implementing it in this project. And That's it. I was ready with the Idea and Technologies to use in my mind and I started searching and learning basics about these techs from here and there.

### Some Recommended Resources To Start Learning Node.Js, Express And MongoDB (Pre-requisite : Vanilla Javascript)

- [Node.Js Tutorial For Beginners | Dev Ed](https://www.youtube.com/watch?v=zQRrXTSkvfw)
- [Build A Restful Api With Node.js Express & MongoDB | Dev Ed](https://www.youtube.com/watch?v=vjf774RKrLc&t=515s)
- [Express Tutorial - MDN Docs](https://developer.mozilla.org/en-US/docs/Learn/Server-side/Express_Nodejs/Tutorial_local_library_website)
- [MongoDB Crash Course | Traversy Media](https://www.youtube.com/watch?v=-56x56UppqQ&t=358s)

___

## Creating Package.json & Installing Dependencies

After all the tutorials and documentation reading, it was time to get my hands dirty and writing some code.

Any node application is generally started by creating a `package.json` file. This file keeps record of all the dependencies used in your application and every information about your project. It is very essential as you cannot upload all your installed node modules online,let's say Github or GitLab. So using this file you can install all the dependencies on the go.

You can create such file by writing following code in the terminal.
```bash
$ npm init
```

Now let's install express and some other useful npm packages by writing the following line in terminal.

```bash
$ npm install express nodemon mongoose
```

[mongoose](https://mongoosejs.com/) is an object modeling package for node.js which helps us to connect to our MongoDB database and create model for our database.
And [nodemon](https://www.npmjs.com/package/nodemon) is a tool that helps develop node.js based applications by automatically restarting the node application when file changes in the directory are detected.

As soon as these packages get installed, you can check in your 'package.json' and you will see that in dependencies these packages along with their installed versions are stored.
___
## Creating Express Server And Writing First Route

After installing the express, I started writing my first route in express in the app.js file.

```javascript
// Importing express in our app
const express = require('express'); 
// Creating express app instance
const app = express.app();

//First Get Route
app.get('/', (req,res) => {
	res.send("Hello World. This is my first route in express!")
});

// Start listening on port 3000.
app.listen(3000, () => console.log('Server Up And Running. Listening On Port 3000') );
```

Now, we can run this script with following code in terminal and it will run just fine.
```bash
$ node app.js
```

But as I've mentioned before, we'll be using nodemon so that we don't have to stop and rerun our server after every change and so now we'll add an npm script in our `package.json` file as below.

```json
"start": "nodemon app.js"
```

And now, all we have to do is write `npm start`
and nodemon will run our server whenever we change any file in our project.

Then, By visiting `http://localhost:3000` I could see my server up and running with my response to the index('/') route as it is.


But we don't want to just send single line responses in our website but we want to send styled html pages to our users.We can do so by Rendering HTML files using many different [template engines](https://expressjs.com/en/guide/using-template-engines.html) such as Jade, EJS, Pug, Mustache, etc. 
___
## Rendering HTML files using EJS template engine

I decided to use EJS because in this application I won't need much complex front-end. To use EJS, first we have to install EJS using npm.

```bash
$ npm install ejs express-ejs-layouts
```
This, express-ejs-layouts package helps us create HTML code blocks and use them in in different HTML files to reduce the code redundancy.For example, we can create a basic layout in which we can create the basic HTML structure and then we don't have to repeat the `head, title, meta, body` tags again in every file.

To use EJS and express-ejs-layouts, we'll have to add following lines in our app.js file
```javascript
const expressLayouts = require('express-ejs-layouts');

app.set('view engine', 'ejs');
app.use(expressLayouts);
```

Now we can render HTML files into our application and using EJS we can also use variables in our HTML code. Now we can write HTML code in our route response but wait, we shouldn't mix our front-end with our back-end in the same file right? Because, that will make debugging task backbreaking and our code will be unreadable.

Also, We won't be writing routes in this app.js files as well for the same reason as above. So now is a good time to restructure our code before it gets unmanagable and tiresome.
___
## Restructuring Project Directory

Directory structure is a very important point in any project and therefore, we will choose the best practice to create our project structure.

Below is the restructured project directory.
```
covid19-updates/
  views/
    index.ejs    
  routes/
    index.js
  app.js
  package.json
```

We'll have to make some changes in our app.js file as well. We'll have to set views directory so that our server will know where to search for our HTML files that we want to render and also we'll have to import route in app.js file.Let's see how our app.js file looks like after these changes.

```javascript
// Importing express in our app
const express = require('express'); 
// Creating express app instance
const app = express.app();

const expressLayouts = require('express-ejs-layouts');

app.set('views',__dirname + '/views');
app.set('view engine', 'ejs');
app.use('expressLayouts');

// To serve Route
require('./routes/index')(app);

// Started listening on port 3000.
app.listen(3000, () => console.log('Server Up And Running. Listening On Port 3000') );
```

And now in routes/index.js file we'll have to export the module that we've created for our index('/') route using module.exports.
_Note: In node.js, every file is treated as a module._ [Read more about modules in node here.](https://nodejs.org/api/modules.html)

```javascript
module.exports = {
	app.get('/'(req,res) => {
		res.render('index.ejs');
	});
}
```

Though, using Router Middleware to serve routes of same prefix is the best practice in an express app. 
_And what in the name of god are Middlewares,now??_ Well, that's a whole another article. For now, we are ready with our index route with  HTML file as response!!

Full code can be found at this repository : 

[Covid-19 Updates India](https://github.com/jamesshah/covid19-updates)


Thank you for reading!! 

#####HappyCoding👩‍💻 👨‍💻

Let's automate the tedious task of Google Form Filling using Python.

How I Automated The Google Form Filling For My College Attendance Using Python

To fight Corona Virus, our Prime Minister has declared lockdown for 21 days in all over the country. And so, our college decided to take online lectures and also, sent us a google form which is to be filled every day for each lecture to fill attendance. Filling form 4 times a day with the subject name and corresponding time and professor details is a tedious task. But also, Attendance is Engineer's Only Motivation to attend lectures. 

And So, I decided to use Python and to write a script to automate this tedious task.


As usual, I started by searching on google to see how can we submit the google form using python script. And I came to know that Every Google Form Input element has a "name" attribute attached to it and we can use it to send the data for the corresponding field.

So, I opened the google form in chrome and Started inspecting Input element in which I will be filling the data, and I found a bunch of _ss-form-entry_ objects with name attribute like `" name = entry.<id> "`

I copied every name attribute value of the field.

Now, In Python I created a dictionary with the value of name attributes that we copied from the browser as keys and the data that I want to fill in the corresponding field as it's value.

```
values = {
            # Email Address
            "emailAddress": str(sys.argv[1]),
            # Enrollment Number
            "entry.33987362": str(sys.argv[2]),
            # Course
            "entry.363926033": "BE",
            # Branch
            "entry.733518766": "IT",
            # Semester
            "entry.114626584": "Sem-6",
            # Subject
            "entry.609979780": i[2],
            # Date
            "entry.1916623197_year": date[0],
            "entry.1916623197_month": date[1],
            "entry.1916623197_day": date[2][0:2],
            # Time
            "entry.125609755_hour": i[0],
            "entry.125609755_minute": i[1],
        }
```

Then I created a dictionary with Day Name(Monday, Tuesday, etc.) as keys and the lectures' details as list as values.

```
subjects_time = {
        "Monday": [["10", "30", "AJ(HC)"], ["11", "30", "DCDR(PC)"], ["1", "00", "WT(AM)"], ["2", "00", "SE(KD)"]],
        "Tuesday": [["11", "30", "SE(KD)"], ["1", "00", "AJ(HC)"]],
        "Wednesday": [["11", "30", ".NET(PC)"], ["1", "00", "SE(KD)"], ["2", "00", ".NET(PC)"]],
        "Thursday": [["10", "30", "WT(AM)"], ["11", "30", "DCDR(PC)"], ["1", "00", "AJ(HC)"]],
        "Friday": [["10", "30", "AJ(HC)"], ["11", "30", ".NET(PC)"], ["1", "00", "SE(KD)"], ["2", "00", ".NET(PC)"]],
    }
```

And put both the dictionaries in a function that returns the list of different data dictionaries of the current day.

```
def get_values():
    """It returns a list of different form data to be submitted by send_attendance method.
    subjects_time is a dictionary with Day as keys and time and subjects in a list as values.
    value_list is a list of lectures' subject and time of current_day."""

    values_list = []
    now = datetime.datetime.now()
    day_name = now.strftime("%A")

    subjects_time = {
        "Monday": [["10", "30", "AJ(HC)"], ["11", "30", "DCDR(PC)"], ["1", "00", "WT(AM)"], ["2", "00", "SE(KD)"]],
        "Tuesday": [["11", "30", "SE(KD)"], ["1", "00", "AJ(HC)"]],
        "Wednesday": [["11", "30", ".NET(PC)"], ["1", "00", "SE(KD)"], ["2", "00", ".NET(PC)"]],
        "Thursday": [["10", "30", "WT(AM)"], ["11", "30", "DCDR(PC)"], ["1", "00", "AJ(HC)"]],
        "Friday": [["10", "30", "AJ(HC)"], ["11", "30", ".NET(PC)"], ["1", "00", "SE(KD)"], ["2", "00", ".NET(PC)"]],
    }

    date = str(now).split('-')

    for i in subjects_time[day_name]:
        '''keys are the value of 'name' element of the '''
        values = {
            # Email Address
            "emailAddress": str(sys.argv[1]),
            # Enrollment Number
            "entry.33987362": str(sys.argv[2]),
            # Course
            "entry.363926033": "BE",
            # Branch
            "entry.733518766": "IT",
            # Semester
            "entry.114626584": "Sem-6",
            # Subject
            "entry.609979780": i[2],
            # Date
            "entry.1916623197_year": date[0],
            "entry.1916623197_month": date[1],
            "entry.1916623197_day": date[2][0:2],
            # Time
            "entry.125609755_hour": i[0],
            "entry.125609755_minute": i[1],
        }

        values_list.append(values)

    return values_list
```


And, finally the function for sending the form response using *POST* method of **requests** library.  It takes two arguments - URL and data(here, data_dictionary), respectively.

```
def send_attendance(url, data):
    """It takes google form url which is to be submitted and also data which is a list of data to be submitted in the form iteratively."""

    for d in data:
        try:
            requests.post(url, data=d)
            print("Form Submitted.")
            time.sleep(5)
        except:
            print("Error Occured!")
```


*Note*: Change the URL of google form by replacing '/viewform'  at the end of the URL with 'formResponse'. Now, your URL should look like this `https://docs.google.com/forms/d/<form_id>/formResponse`

And that's it, now I just have to run this script once a day and it will fill my attendance for all the lectures of that day. 
I can also deploy it on a server like [PythonAnyWhere](https://www.pythonanywhere.com/) or [DigitalOcean](https://www.digitalocean.com/) and it can run by itself every day, but as I'm home quarantine for the 21 Days, I have more than enough time to run it manually every day.🤓🤓

You can find the full code here : [AutoFill Google Form](https://github.com/jamesshah/GoogleForm-AutoFill)
___
Thank Your Reading.#HappyCoding👩‍💻 👨‍💻

9 Helpful Tips For Everyone Interested In Computer Science And Programming

9 Helpful Tips For Every Computer Science Student 

Hello there! In this post, I’m going to share some of the very useful tips that every computer science student should follow. This is not some programming guide but rather these are some general useful tips that you should try to follow if you are a computer science student or if you are planning to learn to program and getting a job in the same. Let’s get started!
___
# 1. Don’t rely on college lectures only, but also don’t bunk all the lectures
You are not going to learn every piece of tech from college lectures, so stop relying on lectures and colleges and start learning on your own but by that, I don’t mean that you should bunk lectures.
Some of the basics of programming and fundamentals you can learn very well from college lectures. But most of the college syllabus has not been updated for the last 10 years or so and therefore it is not so industry-oriented and doesn’t cover the latest technologies. Therefore, you should learn about the latest technologies and frameworks on your own.
___
# 2. Learn how to work with a team
Learn how to communicate with a group of people while building software or working on something. Learn how a project structure is maintained when a bunch of people is working on the same project. Learn version control and git to keep up with the team in a project. Also, learn general communication skills and management skills, it’s very important when you work in corporate culture.

_Bonus Tip_: Work on projects with new people rather than your friends, it will give you the confidence to put your thoughts forward and will push you out of the comfort zone.
___
# 3. Don’t run after learning many languages, learn fundamentals of programming

Learning new programming languages is fun and also a good choice when you are a student because once you start working in a company on a particular project, you won’t get the chance or motivation to learn new programming languages. But don’t rush into learning new languages.
Instead, learn the core concepts of programming. Learn Data Structures and Algorithms, Learn problem-solving, Learn how the program works and how the compilers work. If your basics are clear, learning new languages and frameworks is not an uphill battle.
___
# 4. A good project speaks louder than a bunch of course certificates in your resume

Build projects. In an interview, it doesn’t matter how many courses you have taken and how many certificates you have but what matters is how many projects you have on your resume. And by project, please don’t misunderstand yourself with the To-Do list app or some game. Build real-world like projects. Build a CRUD(Create, Read, Update, Delete) application. Manage databases, Learn to consume APIs and write codes in such a way that others can also read and understand.

_Bonus tip_: Deploy your application online. A live working application won’t just speak louder, it will scream for you!
___
# 5. Build and maintain GitHub and LinkedIn profiles, Do it now!
**THIS IS MUST.** If you are looking forward to getting a job after your college or even after a master’s degree, build a good GitHub and LinkedIn profile. GitHub is your portfolio for your projects and LinkedIn helps you to connect with the Companies and fellow programmers, too.
Upload and maintain your projects in Github repositories, Contribute to Open Source Projects.
Build connections on LinkedIn. It won’t look intimidating at first sight, but it will help you for sure in the long run.
___
# 6. Be patient… You can’t learn it all in a day or a week
This is almost, every newbie programmer ever. You can’t be a Data Scientist in a Day, neither can you build a new Netflix or Instagram in a month.
Learning takes time. Spend 2 to 3 months in learning and practicing the basics of any new technology or framework. Then, build a project, it will teach many more things than the 2–3 months of basics but to start building a project, you must learn those basics. And don’t just know about the languages and frameworks, Understand it. If you have built a complete project, try to optimize the code that you’ve written and also search for the different approaches that you can try to solve that problem.

> “Patience is bitter, but its fruit is sweet.”

___
# 7. Stay updated with the latest trends and technologies

Google about new trends in the industry, Read and join the discussions on online communities. Learn about upcoming technologies in news and articles. Follow a tech YouTube channel or Facebook Page. Share and discuss these topics with your friends.
Also, Participate in online as well as offline Hackathons and coding challenges and meetups. You will meet many fellow programmers and students and will learn a lot from them. Ask them about what they are currently working on and if you find something interesting, learn more about it and share your thoughts on the subjects that you are already familiar with.

For instance, you can follow [ByteTales](https://medium.com/byte-tales) if you want.😉
___
# 8. Do internships, Do a lot of internships!!
You can learn a lot more from an internship than any college lecture or online course. Internships will teach you what it is to be a software developer and an employee in the corporate world.
By working on a big-scale project, You will gain experience of how things work in real life, how to approach a problem and solve it with the team and sometimes by yourself, too. You will get insights into how user management is done in real-life applications.
Internships and Letter Of Recommendation will help you if you in getting into a Master’s Degree in Computer Science. Internships are also very helpful in landing jobs, too but more than that it helps you grow as a developer as well as an employee in general.
___
# 9. Enjoy the journey!!

And last but not least, enjoy college life and have fun. Future and career are important but to chill is important, too. Hangout with friends, Binge watch Tv shows, Travel a lot and Live your life to the fullest. Don’t waste your precious time doing drugs and getting high, it feels cool and great but eventually, you will regret it.

> “Live as if you were to die tomorrow, 
Learn as if you were to live forever.” 
                                                     - Mahatma Gandhi

___

That’s all for now. If you found this post helpful, share it with your friends.
#HappyCoding!💻🙌

Let's create a classic TicTacToe game in Python.

The Classic Tic-Tac-Toe Game in Python 3

# Hello There My Gorgeous Friends On The Internet!

So, the best and the most fun way to learn any programming language for me has always been by developing a fun project like a simple game or some project that I would use in my daily life.

So, when I started to learn Python, I started with this No Starch Press published book[“Automate The Boring Stuff With Python”](https://amzn.to/2piGghV)which is just awesome and If you are looking for a book to get started learning python, I would recommend you to go through this book. Its very beginner-friendly and it covers almost all the basic topics of python. So, while solving the exercises in this book, I came across this TicTacToe game implementation in python.
***
# What we are going to do?
We are going to build a two-player tic-tac-toe game, which we can play in the command-line. Initially, we’ll make an empty game board and then we’ll take inputs from the players and we’ll check for the winning condition and if the whole board gets filled and no one wins, we’ll declare the result as “Tie” and ask users if they want to restart the game.

# What will we use?
We will build this game using Python 3, so make sure you have it installed on your laptop/computer and we are good to go.

# What we’ll learn?
After building this game, we can get a pretty clear idea about dictionaries in python, how to access dictionaries, how to iterate over dictionaries, for loop, if-else conditions and functions in python.

# How does the game work?
The board is numbered like the keyboard’s number pad. And thus, a player can make their move in the game board by entering the number from the keyboard number pad.

# Code Time💻

First, let’s see how we are going to use a dictionary to create our game board. A dictionary is a primitive data type in python which stores data in “key: value” format. and thus, we’ll create a dictionary of length 9 and each key will represent a block in the board and its corresponding value will represent the move made by a player. and we’ll create a function `printBoard()` which we can use every time we want to print the updated board in the game.

```python
theBoard = {'7': ' ' , '8': ' ' , '9': ' ' ,
            '4': ' ' , '5': ' ' , '6': ' ' ,
            '1': ' ' , '2': ' ' , '3': ' ' }
            
def printBoard(board):
    print(board['7'] + '|' + board['8'] + '|' + board['9'])
    print('-+-+-')
    print(board['4'] + '|' + board['5'] + '|' + board['6'])
    print('-+-+-')
    print(board['1'] + '|' + board['2'] + '|' + board['3'])
```

```
Initially our game board will look like this:
 | |
-+-+-
 | |
-+-+-
 | |
```

Now, in the main function, we’ll first take the input from the player and check if the input is a valid move or not. If the block that player requests to move to is valid, we’ll fill that block else we’ll ask the user to choose another block.

```python
def game():

    turn = 'X'
    count = 0
    
    for i in range(10):
            printBoard(theBoard)
            print("It's your turn," + turn + ".Move to which place?")

            move = input()        

            if theBoard[move] == ' ':
                theBoard[move] = turn
                count += 1
            else:
                print("That place is already filled.\nMove to which place?")
                continue
```

Now, to check the winning condition, we’ll check a total of 8 conditions and whichever player has made the last move, we’ll declare that player as a winner. And if no one wins, we’ll declare ‘tie’

```python
# Now we will check if player X or O has won,for every move after 5 moves. 
        if count >= 5:
            if theBoard['7'] == theBoard['8'] == theBoard['9'] != ' ': # across the top
                printBoard(theBoard)
                print("\nGame Over.\n")                
                print(" **** " +turn + " won. ****")                
                break
            elif theBoard['4'] == theBoard['5'] == theBoard['6'] != ' ': # across the middle
                printBoard(theBoard)
                print("\nGame Over.\n")                
                print(" **** " +turn + " won. ****")
                break
            elif theBoard['1'] == theBoard['2'] == theBoard['3'] != ' ': # across the bottom
                printBoard(theBoard)
                print("\nGame Over.\n")                
                print(" **** " +turn + " won. ****")
                break
            elif theBoard['1'] == theBoard['4'] == theBoard['7'] != ' ': # down the left side
                printBoard(theBoard)
                print("\nGame Over.\n")                
                print(" **** " +turn + " won. ****")
                break
            elif theBoard['2'] == theBoard['5'] == theBoard['8'] != ' ': # down the middle
                printBoard(theBoard)
                print("\nGame Over.\n")                
                print(" **** " +turn + " won. ****")
                break
            elif theBoard['3'] == theBoard['6'] == theBoard['9'] != ' ': # down the right side
                printBoard(theBoard)
                print("\nGame Over.\n")                
                print(" **** " +turn + " won. ****")
                break 
            elif theBoard['7'] == theBoard['5'] == theBoard['3'] != ' ': # diagonal
                printBoard(theBoard)
                print("\nGame Over.\n")                
                print(" **** " +turn + " won. ****")
                break
            elif theBoard['1'] == theBoard['5'] == theBoard['9'] != ' ': # diagonal
                printBoard(theBoard)
                print("\nGame Over.\n")                
                print(" **** " +turn + " won. ****")
                break 

        # If neither X nor O wins and the board is full, we'll declare the result as 'tie'.
        if count == 9:
            print("\nGame Over.\n")                
            print("It's a Tie!!")

        # we have to change the player after every move.
        if turn =='X':
            turn = 'O'
        else:
            turn = 'X' 
```

And now, we’ll ask the players if they want to play again.

```python
board_keys = []

for key in theBoard:
    board_keys.append(key)
    
restart = input("Do want to play Again?(y/n)")

    if restart == "y" or restart == "Y":  
        for key in board_keys:
            theBoard[key] = " "

        game()    
```

And boom!! Now we have our game ready.😍😍
***

# Play Time:
```
 | |
-+-+-
 | |
-+-+-
 | |
It's your turn,X.Move to which place?
7
X| |
-+-+-
 | |
-+-+-
 | |
It's your turn,O.Move to which place?
9
X| |O
-+-+-
 | |
-+-+-
 | |
It's your turn,X.Move to which place?
5
X| |O
-+-+-
 |X|
-+-+-
 | |
It's your turn,O.Move to which place?
3
X| |O
-+-+-
 |X|
-+-+-
 | |O
It's your turn,X.Move to which place?
6
X| |O
-+-+-
 |X|X
-+-+-
 | |O
It's your turn,O.Move to which place?
1
X| |O
-+-+-
 |X|X
-+-+-
O| |O
It's your turn,X.Move to which place?
4
X| |O
-+-+-
X|X|X
-+-+-
O| |O

Game Over.

**** X won. ****
```

Thanks For Scrolling, Hope you liked it. #HappyCoding👨‍💻

Let's automate our project initialization process using Python.

Automate Your Project Initialization Using Python & Shell Script

# Hello There My Gorgeous Friends On The Internet.

So every time I start a new project, I repeat the following steps:(most of you would, too)

1. Create a directory for the project.
2. Create and initialize a git repository for that directory.
3. Push this directory onto [Github](https://github.com/jamesshah).
4. Open this project directory in VS Code Editor.

And repeating all these steps for every project is tedious. So as a beginner in python scripting, I thought of automating this whole process using a python script and then executing this script in the command line using shell script.

So what this script does is it asks for a name for your desired project directory and makes a directory inside a directory named ‘Projects’.*(P.S- you can create your directory at your desired location. I just like my projects to be in a single directory. I know, nerd problems!)*And then it initializes a new*git repository*for this directory and pushes onto your Github profile.

> “Talk is cheap, show me the code.”
>
> \-Linus Torvalds

So here it is.

```python
import sys
import os
from github import Github

path = "/path/to/your/desired/projects/directory"

def create():
    folder_name = str(sys.argv[1])
    os.makedirs(path+folder_name)
    username = #your github username
    password = #your github password/ssh key.
    user = Github(username,password).get_user()
    repo = user.create_repo(folder_name)
    print(f"New Repository '{folder_name}' created successfully.")

if __name__ == "__main__":
    create()
```

Just change the path in the above script to your desired path and you add your Github username and password into username and password variables, respectively.*(using an environment variable to hide your password from the script is advised.)*

*You’re still with me, right? Cool*, So now we’ll create a shell script which will add a command ‘create’ and then takes the project directory name as a*command-line argument.*

```shell
#!/bin/bash
#shell script to automate project initialization.
function create(){
    cd  
    python ./automation1.py $1
    cd path/to/your/projects/directory/$1
    touch Readme.md
    git init
    git add .
    git commit -m "first commit"
    git remote add origin https://github.com/<your_git_username>/$1.git
    git push -u origin master
    code . #optional - opens up new directory in vs code editor
}
```

All we are doing is executing the python script that we wrote above using a command ‘create’ and then executing some bash commands in the background using this shell script.

Run the below command in terminal to use this script globally.

`<path/to/this/file>/project_automation.sh`

And now run`create project_name`in your terminal and enjoy your automation.🤘

Thanks For Scrolling, Hope You liked it.  #HappyCoding