dowsery - Text-Summarization-with-Amazon-Reviews-master - SOURCE CODE

# coding: utf-8

# # Summarizing Text with Amazon Reviews

# The objective of this project is to build a model that can create relevant summaries for reviews written about fine foods sold on Amazon. This dataset contains above 500,000 reviews, and is hosted on [Kaggle.](https://www.kaggle.com/snap/amazon-fine-food-reviews)
# 
# To build our model we will use a two-layered bidirectional RNN with LSTMs on the input data and two layers, each with an LSTM using bahdanau attention on the target data. [Jaemin Cho's tutorial](https://github.com/j-min/tf_tutorial_plus/tree/master/RNN_seq2seq/contrib_seq2seq) for seq2seq was really helpful to get the code in working order because this is my first project with TensorFlow 1.1; some of the functions are very different from 1.0. The architecture for this model is similar to Xin Pan's and Peter Liu's, here's their [GitHub page.](https://github.com/tensorflow/models/tree/master/textsum)
# 
# The sections of this project are:
# - Inspecting the Data
# - Preparing the Data
# - Building the Model
# - Training the Model
# - Making Our Own Summaries

# In[49]:

import pandas as pd
import numpy as np
import tensorflow as tf
import re
from nltk.corpus import stopwords
import time
from tensorflow.python.layers.core import Dense
from tensorflow.python.ops.rnn_cell_impl import _zero_state_tensors
print('TensorFlow Version: {}'.format(tf.__version__))

### Cinvert reviews.csv to UTF8 with Notepad
# ## Insepcting the Data

# In[2]:
# https://stackoverflow.com/questions/18039057/python-pandas-error-tokenizing-data
#https://www.tensorflow.org/versions/r1.1/install/install_linux#the_url_of_the_tensorflow_python_package
#https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-1.1.0-cp35-cp35m-linux_x86_64.whl

reviews = pd.read_csv("reviews1.csv", sep=';')


# In[3]:

reviews.shape


# In[4]:

reviews.head()


# In[5]:

... [truncated, 867 more lines] ...

# coding: utf-8

# # Summarizing Text with Amazon Reviews

# The objective of this project is to build a model that can create relevant summaries for reviews written about fine foods sold on Amazon. This dataset contains above 500,000 reviews, and is hosted on [Kaggle.](https://www.kaggle.com/snap/amazon-fine-food-reviews)
# 
# To build our model we will use a two-layered bidirectional RNN with LSTMs on the input data and two layers, each with an LSTM using bahdanau attention on the target data. [Jaemin Cho's tutorial](https://github.com/j-min/tf_tutorial_plus/tree/master/RNN_seq2seq/contrib_seq2seq) for seq2seq was really helpful to get the code in working order because this is my first project with TensorFlow 1.1; some of the functions are very different from 1.0. The architecture for this model is similar to Xin Pan's and Peter Liu's, here's their [GitHub page.](https://github.com/tensorflow/models/tree/master/textsum)
# 
# The sections of this project are:
# - Inspecting the Data
# - Preparing the Data
# - Building the Model
# - Training the Model
# - Making Our Own Summaries

# In[49]:

import pandas as pd
import numpy as np
import tensorflow as tf
import re
from nltk.corpus import stopwords
import time
from tensorflow.python.layers.core import Dense
from tensorflow.python.ops.rnn_cell_impl import _zero_state_tensors
print('TensorFlow Version: {}'.format(tf.__version__))

### Cinvert reviews.csv to UTF8 with Notepad
# ## Insepcting the Data

def encodeToUTF8Adv(text):
	encResults = text.encode('utf-8', "ignore")
	#return str(encResults.decode('latin-1', "ignore"))
	return str(encResults.decode('utf-8', "remove"))

def encodeToLatin1(text):
	#n_String=replaceUmlauts(text)
	encResults = text.encode('utf-8', "ignore")
	#encResults = text.encode('utf-8', "ignore")
	return str(encResults.decode('latin-1', "ignore"))

# In[2]:
# https://stackoverflow.com/questions/18039057/python-pandas-error-tokenizing-data
#https://www.tensorflow.org/versions/r1.1/install/install_linux#the_url_of_the_tensorflow_python_package
#https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-1.1.0-cp35-cp35m-linux_x86_64.whl

reviews = pd.read_csv("reviews1.csv", sep=';')



... [truncated, 899 more lines] ...