Learning-based techniques, especially advanced pre-trained models for code have demonstrated capabilities in code understanding and generation, solving diverse software engineering (SE) tasks. Despite the promising results, current training approaches may not fully optimize model performance, as they typically involve learning from randomly shuffled training data. Recent work shows that Curriculum Learning (CL) can improve NLP task performance through incremental learning based on the difficulty level. Yet, the effectiveness of CL in automated SE tasks remains largely unexplored. In this study, we explore the potential of CL to improve the performance of the pre-trained code model (CodeT5). We assess the effectiveness of CL for fine-tuning the CodeT5 model on two SE tasks: code clone detection and code summarization. We explore two code metrics: code length and cyclomatic complexity to determine the difficulty levels. Our empirical study on the CodeXGLUE benchmark shows that CL has little impact on the model performance for both SE tasks. Surprisingly, the model performance saturates after training on only the first quarter of training data. These results suggest that CL may not provide benefits of improving model performance for SE tasks. This may be due to the model learning capacity. Future work should further explore various CL strategies with various code models for a wider range of SE tasks for a holistic understanding.