Unable to split text received as POST request in django

i am facing a weird error. I am trying to split text sent via form in django. Here is my code in django

from transformers import AutoTokenizer
from langchain.text_splitter import CharacterTextSplitter

tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased-finetuned-sst-2-english" )
text_splitter = CharacterTextSplitter.from_huggingface_tokenizer(
    tokenizer, chunk_size=256, chunk_overlap=0
)

def home(request):
    if request.method == 'POST':
        input_text = request.POST['input']  # Get input text from the POST request
        print("input_text",type(input_text))
        splitted_text = text_splitter.split_text(str(input_text))
        print("splitted_text_length_outside",len(splitted_text))

The length is always 1, which mean text is not split. I have checked that i am receiving text from html form and the type of text is str.

But when i use the same code outside of django such as in jupyter notebook, it work well and split the text.

In django I tried input_text.split() and that worked. But i am clueless why the langchain text splitter is not working.

def home(input_text):
    splitted_text = text_splitter.split_text(input_text)
    print("splitted_text_length_outside",len(splitted_text))

Here is my html form

<form action="{% url 'home' %}" method="post" id="myForm">
{% csrf_token %}
<textarea name="input" id="input" rows="4" cols="50"></textarea>
<br>
<button type="submit">Submit</button>
</form>

Here is my ajax code

$("#myForm").submit(function(event) {
  event.preventDefault();

  let formData = $(this).serialize();

  $.ajax({
    "url": "/",            
    "type": "POST",       
    "data": formData,     
    "success": function(response) {
   
      console.log("success");
    },
    "error": function(error) {
      console.log("error",error);
    }
  });
});

Back to Top